[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:51 dist-test-slave-gkrl krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:51 dist-test-slave-gkrl krb5kdc[399](info): set up 2 sockets
Sep 27 02:43:51 dist-test-slave-gkrl krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 02:43:52.984746 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:53.106465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35921
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:53.211076 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:53.211297 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:53.211335 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:53.215253 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:53.215303 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:53.215322 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:53.215364 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:53.215399 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:53.220489 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:53.221454 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:53.222409 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:53.230528 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:53.230680 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:53.230739 412 server_base.cc:1034] running on GCE node
W20260927 02:43:53.230765 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:53.231343 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:53.232471 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:53.233673 412 hybrid_clock.cc:648] HybridClock initialized: now 1790477033233604 us; error 80 us; skew 500 ppm
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:53.236294 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:53.237155 412 webserver.cc:460] Webserver started at http://127.0.98.62:44685/ using document root <none> and password file <none>
I20260927 02:43:53.237686 412 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:53.237774 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:53.237991 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:53.239933 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "49df201b3e0c4c30a50266aef031dd92"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "5c6d71c9bf3e8a0c56e800d5a46fd0c5"
server_key_iv: "7df66d206eff730b2e961b13aa3ac7d6"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.240505 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "49df201b3e0c4c30a50266aef031dd92"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "5c6d71c9bf3e8a0c56e800d5a46fd0c5"
server_key_iv: "7df66d206eff730b2e961b13aa3ac7d6"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.244174 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:43:53.246362 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.247474 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:53.247617 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "49df201b3e0c4c30a50266aef031dd92"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "5c6d71c9bf3e8a0c56e800d5a46fd0c5"
server_key_iv: "7df66d206eff730b2e961b13aa3ac7d6"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.247743 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:53.268563 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:53.271292 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:53.271520 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:53.278192 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35921
I20260927 02:43:53.278213 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35921 every 8 connection(s)
I20260927 02:43:53.279067 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:53.281679 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:53.282188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 412
I20260927 02:43:53.282346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:53.283561 392 external_mini_cluster.cc:1428] Setting key 76475be39514a0267cc22aff8e45faef
I20260927 02:43:53.287758 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92: Bootstrap starting.
I20260927 02:43:53.289925 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:53.290642 480 log.cc:824] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:53.292433 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92: No bootstrap required, opened a new log
I20260927 02:43:53.295639 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49df201b3e0c4c30a50266aef031dd92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260927 02:43:53.295866 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:53.295961 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49df201b3e0c4c30a50266aef031dd92, State: Initialized, Role: FOLLOWER
I20260927 02:43:53.296370 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49df201b3e0c4c30a50266aef031dd92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260927 02:43:53.296514 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:53.296608 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:53.296797 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:53.297756 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49df201b3e0c4c30a50266aef031dd92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260927 02:43:53.298100 480 leader_election.cc:302] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49df201b3e0c4c30a50266aef031dd92; no voters:
I20260927 02:43:53.298461 480 leader_election.cc:288] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:53.298563 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:53.298797 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [term 1 LEADER]: Becoming Leader. State: Replica: 49df201b3e0c4c30a50266aef031dd92, State: Running, Role: LEADER
I20260927 02:43:53.299129 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49df201b3e0c4c30a50266aef031dd92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260927 02:43:53.299826 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:53.300626 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49df201b3e0c4c30a50266aef031dd92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49df201b3e0c4c30a50266aef031dd92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } } }
I20260927 02:43:53.300738 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [sys.catalog]: This master's current role is: LEADER
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:53.301083 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49df201b3e0c4c30a50266aef031dd92. Latest consensus state: current_term: 1 leader_uuid: "49df201b3e0c4c30a50266aef031dd92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49df201b3e0c4c30a50266aef031dd92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } } }
I20260927 02:43:53.301186 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:53.302142 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:53.305188 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:53.307631 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:53.286486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.0.98.62:35921)
0927 02:43:53.286951 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:53.286975 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:53.288051 (+ 1076us) server_negotiation.cc:407] Connection header received
0927 02:43:53.291039 (+ 2988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:53.291056 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:53.291325 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:53.291635 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:53.293698 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.294631 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.295763 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.296458 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.296663 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:53.302220 (+ 5557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:53.302229 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:53.302247 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:53.302283 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:53.304181 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:53.304699 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:53.304704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:53.304708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:53.304812 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:53.305154 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:53.305157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:53.305159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:53.305582 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:53.305790 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:53.306330 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 02:43:53.306622 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":194,"threads_started":1}
I20260927 02:43:53.310842 492 catalog_manager.cc:1347] Generated new cluster ID: bfacde9566f240c792565b81c5f872c0
I20260927 02:43:53.310913 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:53.329638 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:53.330833 492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:53.336994 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49df201b3e0c4c30a50266aef031dd92: Generated new TSK 0
I20260927 02:43:53.337775 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:43:53.367071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:53.493191 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:53.493436 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:53.493479 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:53.497413 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:53.497466 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:53.497524 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:43:53.502980 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:53.503971 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:53.504918 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:53.514122 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:53.514154 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:53.514195 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:53.514523 506 server_base.cc:1034] running on GCE node
I20260927 02:43:53.514875 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:53.515470 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:53.516623 506 hybrid_clock.cc:648] HybridClock initialized: now 1790477033516591 us; error 45 us; skew 500 ppm
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:53.519343 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:43:53.520274 506 webserver.cc:460] Webserver started at http://127.0.98.1:43663/ using document root <none> and password file <none>
I20260927 02:43:53.520820 506 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:53.520880 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:53.521111 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:53.523178 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd9ca04e8b4d410a941b9e3d3defcacd"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "3d591f5709fe51d8ab20171487fb5ce4"
server_key_iv: "0c62432c78613ac91a114e074b529390"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.523736 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd9ca04e8b4d410a941b9e3d3defcacd"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "3d591f5709fe51d8ab20171487fb5ce4"
server_key_iv: "0c62432c78613ac91a114e074b529390"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.527300 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:43:53.529649 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.530821 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:53.530954 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "bd9ca04e8b4d410a941b9e3d3defcacd"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "3d591f5709fe51d8ab20171487fb5ce4"
server_key_iv: "0c62432c78613ac91a114e074b529390"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.531061 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:53.553884 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:53.557096 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:53.557315 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:53.557953 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:53.558836 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:53.558884 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.558923 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:53.558939 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.567960 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33871
I20260927 02:43:53.567983 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33871 every 8 connection(s)
I20260927 02:43:53.568889 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:43:53.573632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 506
I20260927 02:43:53.573719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:43:53.573975 392 external_mini_cluster.cc:1428] Setting key 1773357d23d47bf2810a3d3eadd176ce
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:43:53.583794 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:53.570635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58325 (local address 127.0.98.62:35921)
0927 02:43:53.570778 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:53.570783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:53.571632 (+ 849us) server_negotiation.cc:407] Connection header received
0927 02:43:53.573017 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:53.573020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:53.573076 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:53.573172 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:53.574434 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.575086 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.577091 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.577491 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.577618 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:53.580369 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:53.580372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:53.580374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:53.580395 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:53.581808 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:53.582315 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:53.582323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:53.582324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:53.582395 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:53.582739 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:53.582742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:53.582744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:53.582915 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:53.583027 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:53.583493 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 02:43:53.583628 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:43:53.584662 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:53.571022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35921 (local address 127.0.98.1:58325)
0927 02:43:53.571488 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:53.571531 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:53.572805 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:53.573352 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:53.573358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:53.573757 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:53.574292 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:53.574305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.575413 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.575416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:53.576944 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:53.576959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.577658 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.577661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:53.577807 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:53.578450 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:53.578466 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:53.580204 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:53.581934 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:53.581939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:53.581953 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:53.582209 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:53.582488 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:53.582498 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:53.582500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:53.582638 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:53.583025 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:53.583032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:53.583261 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 02:43:53.583438 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 02:43:53.583752 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:43:53.586021 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35921
I20260927 02:43:53.586279 635 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:53.586762 635 heartbeater.cc:511] Master 127.0.98.62:35921 requested a full tablet report, sending...
I20260927 02:43:53.588400 444 ts_manager.cc:194] Registered new tserver with Master: bd9ca04e8b4d410a941b9e3d3defcacd (127.0.98.1:33871)
I20260927 02:43:53.589483 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:43:53.600610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:53.711023 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:53.711282 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:53.711385 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:53.715560 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:53.715633 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:53.715716 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:43:53.720912 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:53.721913 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:53.722887 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:53.732200 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:53.732190 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:53.732296 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:53.732533 640 server_base.cc:1034] running on GCE node
I20260927 02:43:53.732932 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:53.733594 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:53.734815 640 hybrid_clock.cc:648] HybridClock initialized: now 1790477033734764 us; error 44 us; skew 500 ppm
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:53.737908 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:43:53.738720 640 webserver.cc:460] Webserver started at http://127.0.98.2:36149/ using document root <none> and password file <none>
I20260927 02:43:53.739215 640 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:53.739271 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:53.739424 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:53.741401 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "7e655051b9a24121bc59bacf34efbce9"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "ca06e52fed3dc3a2711aa8dd15dc8aef"
server_key_iv: "c957e23c4c310d08d72c4585a7e7bf5f"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.741915 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7e655051b9a24121bc59bacf34efbce9"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "ca06e52fed3dc3a2711aa8dd15dc8aef"
server_key_iv: "c957e23c4c310d08d72c4585a7e7bf5f"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.745505 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:43:53.747726 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.749070 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:53.749243 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "7e655051b9a24121bc59bacf34efbce9"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "ca06e52fed3dc3a2711aa8dd15dc8aef"
server_key_iv: "c957e23c4c310d08d72c4585a7e7bf5f"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.749363 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:53.761722 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:53.764791 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:53.765008 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:53.765591 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:53.766464 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:53.766552 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.766628 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:53.766681 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.777550 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37229
I20260927 02:43:53.777575 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37229 every 8 connection(s)
I20260927 02:43:53.778525 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:43:53.786242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 640
I20260927 02:43:53.786372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:43:53.786703 392 external_mini_cluster.cc:1428] Setting key e02ccf05c717e9885b3082f73ff6a0c5
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:53.792630 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:53.780396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49687 (local address 127.0.98.62:35921)
0927 02:43:53.780542 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:53.780546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:53.781409 (+ 863us) server_negotiation.cc:407] Connection header received
0927 02:43:53.782665 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:53.782668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:53.782713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:53.782785 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:53.783899 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.784447 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.785670 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.785877 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.785964 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:53.788556 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:53.788559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:53.788561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:53.788585 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:53.790236 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:53.790849 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:53.790852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:53.790853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:53.790928 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:53.791304 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:53.791328 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:53.791335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:53.791602 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:53.791708 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:53.792265 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 02:43:53.792397 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:43:53.793291 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:53.780835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35921 (local address 127.0.98.2:49687)
0927 02:43:53.781262 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:53.781303 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:53.782458 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:53.782936 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:53.782942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:53.783287 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:53.783749 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:53.783761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.784571 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.784574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:53.785560 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:53.785570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.786015 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.786018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:53.786150 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:53.786765 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:53.786780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:53.788389 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:53.790345 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:53.790354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:53.790369 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:53.790737 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:53.791019 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:53.791024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:53.791026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:53.791205 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:53.791725 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:53.791730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:53.792027 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 02:43:53.792183 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 02:43:53.792415 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260927 02:43:53.794348 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35921
I20260927 02:43:53.794586 769 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:53.795032 769 heartbeater.cc:511] Master 127.0.98.62:35921 requested a full tablet report, sending...
I20260927 02:43:53.796068 444 ts_manager.cc:194] Registered new tserver with Master: 7e655051b9a24121bc59bacf34efbce9 (127.0.98.2:37229)
I20260927 02:43:53.796620 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49687
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:43:53.814254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:53.921743 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:53.922015 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:53.922092 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:53.926067 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:53.926146 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:53.926235 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:43:53.932224 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:53.933202 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:53.934162 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:53.943413 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:53.943423 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:53.943444 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:53.943670 774 server_base.cc:1034] running on GCE node
I20260927 02:43:53.944144 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:53.944767 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:53.945945 774 hybrid_clock.cc:648] HybridClock initialized: now 1790477033945922 us; error 39 us; skew 500 ppm
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:53.948670 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:43:53.949505 774 webserver.cc:460] Webserver started at http://127.0.98.3:41453/ using document root <none> and password file <none>
I20260927 02:43:53.950083 774 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:53.950170 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:53.950431 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:53.952504 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "44ce260e9afb4de78e3b133105f59e4f"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "f1e2b1a3243459dce4b68abebee2e34a"
server_key_iv: "f1d5fd9ae628a6e64cd132f7b4ac061c"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.953132 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "44ce260e9afb4de78e3b133105f59e4f"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "f1e2b1a3243459dce4b68abebee2e34a"
server_key_iv: "f1d5fd9ae628a6e64cd132f7b4ac061c"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.957065 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 02:43:53.960145 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.961279 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:43:53.961400 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "44ce260e9afb4de78e3b133105f59e4f"
format_stamp: "Formatted at 2026-09-27 02:43:53 on dist-test-slave-gkrl"
server_key: "f1e2b1a3243459dce4b68abebee2e34a"
server_key_iv: "f1d5fd9ae628a6e64cd132f7b4ac061c"
server_key_version: "encryptionkey@0"
I20260927 02:43:53.961503 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:53.977349 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:53.980044 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:53.980250 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:53.980806 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:53.981583 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:53.981632 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.981698 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:53.981732 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:53.990937 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37101
I20260927 02:43:53.991008 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37101 every 8 connection(s)
I20260927 02:43:53.991830 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:43:53.999997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 774
I20260927 02:43:54.000154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:43:54.000500 392 external_mini_cluster.cc:1428] Setting key dbc89b890e1e73f6ce9ca09494c8c960
Sep 27 02:43:53 dist-test-slave-gkrl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:54.005342 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:53.993644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59379 (local address 127.0.98.62:35921)
0927 02:43:53.993783 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:53.993788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:53.994553 (+ 765us) server_negotiation.cc:407] Connection header received
0927 02:43:53.995812 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:53.995815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:53.995857 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:53.995987 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:53.997061 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.997589 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.998650 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.998845 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.998916 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:54.001738 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:54.001742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:54.001744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:54.001768 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:54.003353 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:54.003841 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:54.003847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:54.003848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:54.003970 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:54.004299 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:54.004303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:54.004305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:54.004491 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:54.004612 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:54.005056 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 02:43:54.005179 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:43:54.006423 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:53.993991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35921 (local address 127.0.98.3:59379)
0927 02:43:53.994403 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:53.994443 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:53.995614 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:53.996120 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:53.996126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:53.996478 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:53.996932 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:53.996944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.997714 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.997716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:53.998544 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:53.998553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:53.998958 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:53.998961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:53.999089 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:53.999602 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:53.999617 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:54.001563 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:54.003470 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:54.003475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:54.003488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:54.003738 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:54.004068 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:54.004071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:54.004073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:54.004204 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:54.004596 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:54.004604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:54.004835 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 02:43:54.005015 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 02:43:54.005308 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260927 02:43:54.007781 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35921
I20260927 02:43:54.008040 903 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:54.008561 903 heartbeater.cc:511] Master 127.0.98.62:35921 requested a full tablet report, sending...
I20260927 02:43:54.009774 444 ts_manager.cc:194] Registered new tserver with Master: 44ce260e9afb4de78e3b133105f59e4f (127.0.98.3:37101)
I20260927 02:43:54.010514 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59379
I20260927 02:43:54.014559 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:43:54 dist-test-slave-gkrl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:43:54 dist-test-slave-gkrl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:43:54.039714 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:54.030596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.0.98.1:33871)
0927 02:43:54.030899 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:54.030918 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:54.030940 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:43:54.030988 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:54.030993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:54.031173 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:54.031306 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:54.031928 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:54.032672 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:54.033491 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:54.033845 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:54.033952 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:54.036330 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:54.036338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:54.036351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:54.036373 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:54.037941 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:54.038423 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:54.038428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:54.038429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:54.038531 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:54.038758 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:54.038760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:54.038762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:54.038983 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:54.039145 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:54.039351 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 02:43:54.039505 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":115,"threads_started":1}
W20260927 02:43:54.040659 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51244
Sep 27 02:43:54 dist-test-slave-gkrl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:43:54 dist-test-slave-gkrl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:43:54.065008 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:54.057312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.0.98.1:33871)
0927 02:43:54.057482 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:54.057486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:54.057553 (+ 67us) server_negotiation.cc:407] Connection header received
0927 02:43:54.057658 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:54.057661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:54.057704 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:54.057772 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:54.058409 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:54.058913 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:54.059709 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:54.059981 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:54.060059 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:54.062331 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:54.062334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:54.062335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:54.062356 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:54.063543 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:54.063967 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:54.063969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:54.063970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:54.064039 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:54.064268 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:54.064270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:54.064271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:54.064416 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:54.064510 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:54.064712 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 02:43:54.064800 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 02:43:54.066092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 506
I20260927 02:43:54.072002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 640
I20260927 02:43:54.079169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 774
I20260927 02:43:54.084380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 412
2026-09-27T02:43:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2161 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:54 dist-test-slave-gkrl krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:54 dist-test-slave-gkrl krb5kdc[917](info): set up 2 sockets
Sep 27 02:43:54 dist-test-slave-gkrl krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 02:43:55.123415 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:55 dist-test-slave-gkrl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:55.243566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33113
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:55.349321 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:55.349583 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:55.349624 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:55.353737 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:55.353790 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:55.353809 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:55.353826 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:55.353880 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:55.359036 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:55.360119 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:55.361380 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:55.371411 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:55.371411 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:55.371624 930 server_base.cc:1034] running on GCE node
W20260927 02:43:55.371462 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:55.372309 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:55.373437 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:55.374619 930 hybrid_clock.cc:648] HybridClock initialized: now 1790477035374594 us; error 46 us; skew 500 ppm
Sep 27 02:43:55 dist-test-slave-gkrl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:55.378394 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:55.379323 930 webserver.cc:460] Webserver started at http://127.0.98.62:35313/ using document root <none> and password file <none>
I20260927 02:43:55.379974 930 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:55.380044 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:55.380280 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:55.382292 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05"
format_stamp: "Formatted at 2026-09-27 02:43:55 on dist-test-slave-gkrl"
server_key: "26530d03c010e85cc012d583c54dcabf"
server_key_iv: "6ae8089597969c2cd8e6884efe9da137"
server_key_version: "encryptionkey@0"
I20260927 02:43:55.382858 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05"
format_stamp: "Formatted at 2026-09-27 02:43:55 on dist-test-slave-gkrl"
server_key: "26530d03c010e85cc012d583c54dcabf"
server_key_iv: "6ae8089597969c2cd8e6884efe9da137"
server_key_version: "encryptionkey@0"
I20260927 02:43:55.386603 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:43:55.389238 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:55.390569 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:43:55.390719 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05"
format_stamp: "Formatted at 2026-09-27 02:43:55 on dist-test-slave-gkrl"
server_key: "26530d03c010e85cc012d583c54dcabf"
server_key_iv: "6ae8089597969c2cd8e6884efe9da137"
server_key_version: "encryptionkey@0"
I20260927 02:43:55.390832 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:55.410298 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:55.413455 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:55.413662 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:55.421329 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33113 every 8 connection(s)
I20260927 02:43:55.421330 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33113
I20260927 02:43:55.422359 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:55.425661 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:55.429104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 930
I20260927 02:43:55.429219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:55.429466 392 external_mini_cluster.cc:1428] Setting key 0c792729ea3ac276ea38ffa9ef67e095
I20260927 02:43:55.431699 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05: Bootstrap starting.
I20260927 02:43:55.434245 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:55.435109 998 log.cc:824] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:55.437335 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05: No bootstrap required, opened a new log
Sep 27 02:43:55 dist-test-slave-gkrl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:55.442085 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33113 } }
I20260927 02:43:55.442363 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:55.442428 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe66ff8ffe7c4e5d8bb9931bf8c54f05, State: Initialized, Role: FOLLOWER
I20260927 02:43:55.442924 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33113 } }
I20260927 02:43:55.443082 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:55.443162 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:55.443248 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:55.445065 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:55.430804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.0.98.62:33113)
0927 02:43:55.431194 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:55.431218 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:55.431253 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:43:55.431993 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:55.432012 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:55.432348 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:55.432621 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:55.433469 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:55.434516 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:55.435706 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:55.436283 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:55.436403 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:55.439163 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:55.439170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:55.439187 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:55.439217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:55.442001 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:55.442469 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:55.442475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:55.442482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:55.442628 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:55.442940 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:55.442945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:55.442946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:55.443474 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:55.443705 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:55.443997 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 02:43:55.444318 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":122,"threads_started":1}
I20260927 02:43:55.482385 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33113 } }
I20260927 02:43:55.482908 998 leader_election.cc:302] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe66ff8ffe7c4e5d8bb9931bf8c54f05; no voters:
I20260927 02:43:55.483421 998 leader_election.cc:288] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:55.483697 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:55.483932 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [term 1 LEADER]: Becoming Leader. State: Replica: fe66ff8ffe7c4e5d8bb9931bf8c54f05, State: Running, Role: LEADER
I20260927 02:43:55.484305 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33113 } }
I20260927 02:43:55.484862 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:55.486143 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33113 } } }
I20260927 02:43:55.486294 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:55.486718 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe66ff8ffe7c4e5d8bb9931bf8c54f05. Latest consensus state: current_term: 1 leader_uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe66ff8ffe7c4e5d8bb9931bf8c54f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33113 } } }
I20260927 02:43:55.486878 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05 [sys.catalog]: This master's current role is: LEADER
W20260927 02:43:55.490283 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:43:55.490372 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:43:55.490454 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:55.491150 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:55.498057 1012 catalog_manager.cc:1347] Generated new cluster ID: a426c177ed8c49469e5d7f8c544c3817
I20260927 02:43:55.498127 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:55.505584 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:55.506870 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:55.513381 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe66ff8ffe7c4e5d8bb9931bf8c54f05: Generated new TSK 0
I20260927 02:43:55.514149 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260927 02:43:56.524266 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 1.006s user 0.001s sys 0.002s
I20260927 02:43:56.524752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33113
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:56.638568 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:56.638845 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:56.638890 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:56.643026 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:56.643091 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:56.643177 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:43:56.648492 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:56.649559 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:56.650661 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:56.660585 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:56.660578 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:56.660668 1024 server_base.cc:1034] running on GCE node
W20260927 02:43:56.660580 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:56.661312 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:56.662051 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:56.663237 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790477036663222 us; error 31 us; skew 500 ppm
Sep 27 02:43:56 dist-test-slave-gkrl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:56.666075 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:43:56.666877 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39131/ using document root <none> and password file <none>
I20260927 02:43:56.667382 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:56.667438 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:56.667610 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:56.669668 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "d77cfedf03c641e791dac383b2f8cd7b"
format_stamp: "Formatted at 2026-09-27 02:43:56 on dist-test-slave-gkrl"
server_key: "41180af2b4b09c66968fd40edeb9b6bf"
server_key_iv: "db544d4d2160c22b741c1778a5315675"
server_key_version: "encryptionkey@0"
I20260927 02:43:56.670233 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d77cfedf03c641e791dac383b2f8cd7b"
format_stamp: "Formatted at 2026-09-27 02:43:56 on dist-test-slave-gkrl"
server_key: "41180af2b4b09c66968fd40edeb9b6bf"
server_key_iv: "db544d4d2160c22b741c1778a5315675"
server_key_version: "encryptionkey@0"
I20260927 02:43:56.674080 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 02:43:56.676609 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:56.677836 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:43:56.677966 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "d77cfedf03c641e791dac383b2f8cd7b"
format_stamp: "Formatted at 2026-09-27 02:43:56 on dist-test-slave-gkrl"
server_key: "41180af2b4b09c66968fd40edeb9b6bf"
server_key_iv: "db544d4d2160c22b741c1778a5315675"
server_key_version: "encryptionkey@0"
I20260927 02:43:56.678076 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:56.699278 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:56.702335 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:56.702575 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:56.703384 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:56.704594 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:56.704668 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:56.704730 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:56.704767 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:56.715528 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39317
I20260927 02:43:56.715610 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39317 every 8 connection(s)
I20260927 02:43:56.716530 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:43:56.721339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 1024
I20260927 02:43:56.721457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:43:56.721714 392 external_mini_cluster.cc:1428] Setting key 6b3220d89e9ab64cbca5fe24f4939c95
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 02:43:56 dist-test-slave-gkrl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:56.735144 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:56.718342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53013 (local address 127.0.98.62:33113)
0927 02:43:56.718592 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:56.718596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:56.719302 (+ 706us) server_negotiation.cc:407] Connection header received
0927 02:43:56.720767 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:56.720771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:56.720834 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:56.720945 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:56.722315 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:56.723094 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:56.725557 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:56.725993 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:56.726127 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:56.731848 (+ 5721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:56.731852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:56.731854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:56.731936 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:56.733321 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:56.733801 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:56.733804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:56.733805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:56.733907 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:56.734192 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:56.734194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:56.734196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:56.734365 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:56.734469 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:56.734912 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 02:43:56.735010 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260927 02:43:56.736092 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:56.718614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33113 (local address 127.0.98.1:53013)
0927 02:43:56.719123 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:56.719175 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:56.720540 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:56.721116 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:56.721125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:56.721582 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:56.722150 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:56.722181 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:56.723230 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:56.723234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:56.725428 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:56.725438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:56.726161 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:56.726164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:56.726337 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:56.729879 (+ 3542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:56.729897 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:56.731634 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:56.733423 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:56.733428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:56.733441 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:56.733710 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:56.733981 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:56.733983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:56.733985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:56.734110 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:56.734452 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:56.734456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:56.734704 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 02:43:56.734855 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 02:43:56.735103 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":125,"threads_started":1}
I20260927 02:43:56.737355 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33113
I20260927 02:43:56.737589 1153 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:56.738029 1153 heartbeater.cc:511] Master 127.0.98.62:33113 requested a full tablet report, sending...
I20260927 02:43:56.739483 959 ts_manager.cc:194] Registered new tserver with Master: d77cfedf03c641e791dac383b2f8cd7b (127.0.98.1:39317)
I20260927 02:43:56.740641 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:43:57.742882 1153 heartbeater.cc:503] Master 127.0.98.62:33113 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 02:43:57.870965 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.147s user 0.000s sys 0.003s
I20260927 02:43:57.871532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33113
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:57.987152 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:57.987397 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:57.987471 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:57.991842 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:57.991945 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:57.992044 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:43:57.997666 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:57.998792 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:57.999831 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:58.010913 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:58.010902 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:58.010905 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:58.011897 1159 server_base.cc:1034] running on GCE node
I20260927 02:43:58.012387 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:58.013209 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:58.014391 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790477038014356 us; error 46 us; skew 500 ppm
Sep 27 02:43:58 dist-test-slave-gkrl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:58.020008 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:43:58.020936 1159 webserver.cc:460] Webserver started at http://127.0.98.2:46693/ using document root <none> and password file <none>
I20260927 02:43:58.021494 1159 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:58.021610 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:58.021834 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:58.024046 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "d1dfb0f59fe54de7b8f605e2d1671076"
format_stamp: "Formatted at 2026-09-27 02:43:58 on dist-test-slave-gkrl"
server_key: "37b733a867c4e875b9a55361fb5f7ee3"
server_key_iv: "a8987ed60db63e8302a9b3ada0252edf"
server_key_version: "encryptionkey@0"
I20260927 02:43:58.024684 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d1dfb0f59fe54de7b8f605e2d1671076"
format_stamp: "Formatted at 2026-09-27 02:43:58 on dist-test-slave-gkrl"
server_key: "37b733a867c4e875b9a55361fb5f7ee3"
server_key_iv: "a8987ed60db63e8302a9b3ada0252edf"
server_key_version: "encryptionkey@0"
I20260927 02:43:58.028532 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 02:43:58.030944 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:58.032294 1159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:58.032428 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "d1dfb0f59fe54de7b8f605e2d1671076"
format_stamp: "Formatted at 2026-09-27 02:43:58 on dist-test-slave-gkrl"
server_key: "37b733a867c4e875b9a55361fb5f7ee3"
server_key_iv: "a8987ed60db63e8302a9b3ada0252edf"
server_key_version: "encryptionkey@0"
I20260927 02:43:58.032534 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:58.059865 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:58.063393 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:58.063601 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:58.064248 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:58.065120 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:58.065186 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:58.065255 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:58.065306 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:58.074340 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33333
I20260927 02:43:58.074404 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33333 every 8 connection(s)
I20260927 02:43:58.075361 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:43:58.077926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 1159
I20260927 02:43:58.078063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:43:58.078372 392 external_mini_cluster.cc:1428] Setting key 1d9d19824deec25f938f794bd17554c9
Sep 27 02:43:58 dist-test-slave-gkrl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:43:58.091748 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.077335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54805 (local address 127.0.98.62:33113)
0927 02:43:58.077623 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:58.077628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:58.078398 (+ 770us) server_negotiation.cc:407] Connection header received
0927 02:43:58.080038 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:58.080043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:58.080102 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:58.080194 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:58.081833 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.082442 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.083931 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.084356 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.084481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.087651 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:58.087655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:58.087657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:58.087680 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:58.089631 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:58.090200 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:58.090205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:58.090206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:58.090296 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:58.090631 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:58.090636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:58.090638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:58.090825 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:58.090932 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:58.091427 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 02:43:58.091565 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260927 02:43:58.092555 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.077780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33113 (local address 127.0.98.2:54805)
0927 02:43:58.078246 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:58.078299 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:58.079805 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:58.080394 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:58.080404 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:58.080941 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:58.081650 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.081671 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.082575 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.082582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.083787 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.083796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.084581 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.084585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.084747 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.085414 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:58.085440 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:58.087474 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:58.089753 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:58.089760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:58.089783 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:58.090088 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:58.090389 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:58.090394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:58.090396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:58.090535 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:58.090929 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:58.090936 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:58.091192 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 02:43:58.091387 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 02:43:58.091709 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":145,"threads_started":1}
I20260927 02:43:58.093739 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33113
I20260927 02:43:58.093955 1288 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:58.094493 1288 heartbeater.cc:511] Master 127.0.98.62:33113 requested a full tablet report, sending...
I20260927 02:43:58.095419 959 ts_manager.cc:194] Registered new tserver with Master: d1dfb0f59fe54de7b8f605e2d1671076 (127.0.98.2:33333)
I20260927 02:43:58.096035 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260927 02:43:58.731662 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.651s user 0.002s sys 0.002s
I20260927 02:43:58.732424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33113
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:58.846539 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:58.846818 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:58.846889 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:58.851493 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:58.851573 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:58.851666 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:43:58.856927 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:58.857936 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:58.859063 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:58.868779 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:58.868901 1294 server_base.cc:1034] running on GCE node
W20260927 02:43:58.868800 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:58.869086 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:58.869470 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:58.870090 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:58.871269 1294 hybrid_clock.cc:648] HybridClock initialized: now 1790477038871248 us; error 32 us; skew 500 ppm
Sep 27 02:43:58 dist-test-slave-gkrl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:58.874022 1294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:43:58.874845 1294 webserver.cc:460] Webserver started at http://127.0.98.3:33689/ using document root <none> and password file <none>
I20260927 02:43:58.875391 1294 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:58.875500 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:58.875732 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:58.877700 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7"
format_stamp: "Formatted at 2026-09-27 02:43:58 on dist-test-slave-gkrl"
server_key: "3204089ac9f82272ebff156a21e092e5"
server_key_iv: "31ec341e3b48fc09c624a76143a3e99c"
server_key_version: "encryptionkey@0"
I20260927 02:43:58.878274 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7"
format_stamp: "Formatted at 2026-09-27 02:43:58 on dist-test-slave-gkrl"
server_key: "3204089ac9f82272ebff156a21e092e5"
server_key_iv: "31ec341e3b48fc09c624a76143a3e99c"
server_key_version: "encryptionkey@0"
I20260927 02:43:58.881970 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:43:58.884783 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:58.886088 1294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:58.886228 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7"
format_stamp: "Formatted at 2026-09-27 02:43:58 on dist-test-slave-gkrl"
server_key: "3204089ac9f82272ebff156a21e092e5"
server_key_iv: "31ec341e3b48fc09c624a76143a3e99c"
server_key_version: "encryptionkey@0"
I20260927 02:43:58.886336 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:58.902060 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:58.905246 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:58.905441 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:58.905997 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:58.906767 1294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:58.906834 1294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:58.906904 1294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:58.906947 1294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:58.915757 1294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38111
I20260927 02:43:58.915783 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38111 every 8 connection(s)
I20260927 02:43:58.916620 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:43:58.919248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 1294
I20260927 02:43:58.919369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:43:58.919682 392 external_mini_cluster.cc:1428] Setting key 182e22b0e3d20858c1d53f400bcab8cf
Sep 27 02:43:58 dist-test-slave-gkrl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:58.931476 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.918477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47647 (local address 127.0.98.62:33113)
0927 02:43:58.918761 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:58.918769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:58.919517 (+ 748us) server_negotiation.cc:407] Connection header received
0927 02:43:58.921552 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:58.921555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:58.921601 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:58.921732 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:58.923135 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.923682 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.924894 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.925131 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.925246 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.927734 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:58.927738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:58.927740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:58.927771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:58.929379 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:58.929927 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:58.929932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:58.929934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:58.930022 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:58.930355 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:58.930360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:58.930361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:58.930540 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:58.930657 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:58.931167 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 02:43:58.931301 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260927 02:43:58.932332 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.918875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33113 (local address 127.0.98.3:47647)
0927 02:43:58.919347 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:58.919405 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:58.921290 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:58.921919 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:58.921929 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:58.922370 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:58.922984 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.923004 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.923817 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.923821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.924765 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.924778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.925276 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.925280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.925458 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.926099 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:58.926114 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:58.927551 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:58.929508 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:58.929513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:58.929527 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:58.929828 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:58.930119 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:58.930122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:58.930124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:58.930259 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:58.930652 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:58.930658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:58.930941 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 02:43:58.931125 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 02:43:58.931420 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260927 02:43:58.933676 1423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33113
I20260927 02:43:58.933918 1423 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:58.934470 1423 heartbeater.cc:511] Master 127.0.98.62:33113 requested a full tablet report, sending...
I20260927 02:43:58.935428 959 ts_manager.cc:194] Registered new tserver with Master: d6b5f95d9ef844fdbc53b389bf8fecb7 (127.0.98.3:38111)
I20260927 02:43:58.936059 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47647
I20260927 02:43:58.946406 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:43:58.957490 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.950081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.0.98.62:33113)
0927 02:43:58.950242 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:58.950247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:58.950368 (+ 121us) server_negotiation.cc:407] Connection header received
0927 02:43:58.950529 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:58.950532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:58.950583 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:58.950668 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:58.951376 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.951949 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.952771 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.953000 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.953109 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.954453 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:58.954456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:58.954465 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:58.954490 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:58.955807 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:58.956256 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:58.956264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:58.956266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:58.956336 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:58.956604 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:58.956607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:58.956608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:58.956825 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:58.956970 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:58.957221 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 02:43:58.957339 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:43:58.962116 959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:43:58.965257 959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:43:58.979971 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.973536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33333 (local address 127.0.0.1:58902)
0927 02:43:58.974257 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:58.974308 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:58.974456 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:58.975156 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:58.975161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:58.975211 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:58.975349 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.975355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.976768 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.976772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.978088 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.978099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.979434 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.979436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.979530 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.979557 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 02:43:58.979670 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:43:58.979816 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260927 02:43:58.980110 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.973649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.0.98.2:33333)
0927 02:43:58.974007 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:58.974013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:58.974402 (+ 389us) server_negotiation.cc:407] Connection header received
0927 02:43:58.974617 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:58.974624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:58.974827 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:58.974954 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:58.975492 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.976638 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.978228 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.979135 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.979197 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.979297 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:58.979710 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 02:43:58.979808 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":70,"threads_started":1}
I20260927 02:43:58.981437 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.974184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38111 (local address 127.0.0.1:39660)
0927 02:43:58.975099 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:58.975113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:58.975185 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:58.976745 (+ 1560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:58.976748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:58.976767 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:58.977089 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.977096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.978533 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.978538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.979696 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.979706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.980960 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.980964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.981051 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.981075 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:43:58.981145 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:43:58.981223 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":213,"thread_start_us":104,"threads_started":1}
I20260927 02:43:58.981590 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.973636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.0.1:34884)
0927 02:43:58.974257 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:58.974309 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:58.974436 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:58.975667 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:58.975672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:58.975695 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:58.975800 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.975807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.978939 (+ 3132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.978943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.980113 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:58.980122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.981186 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.981193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:58.981278 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.981310 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:43:58.981390 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:43:58.981463 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":25,"thread_start_us":100,"threads_started":1}
I20260927 02:43:58.981659 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.974357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.0.98.1:39317)
0927 02:43:58.974887 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:58.974893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:58.974916 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:43:58.975000 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:58.975007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:58.975341 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:58.975565 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:58.976237 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.978820 (+ 2583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.980245 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.981050 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.981157 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.981286 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:58.981428 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 02:43:58.981529 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":136,"threads_started":1}
I20260927 02:43:58.981858 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:58.974705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.0.98.3:38111)
0927 02:43:58.975250 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:58.975261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:58.975283 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:43:58.976142 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:58.976148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:58.976410 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:58.976839 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:58.977246 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.978370 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.979838 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:58.980645 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:58.980746 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:58.980872 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:58.981587 (+ 715us) server_negotiation.cc:299] Negotiation successful
0927 02:43:58.981676 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":133,"threads_started":1}
I20260927 02:43:58.983709 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8ec37e4df4a6a88c09e92952935e3 (DEFAULT_TABLE table=test-table [id=f1ab5420a24f4af1b7aa81005264280b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:58.984613 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8ec37e4df4a6a88c09e92952935e3 (DEFAULT_TABLE table=test-table [id=f1ab5420a24f4af1b7aa81005264280b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:58.984977 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a8ec37e4df4a6a88c09e92952935e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:58.984637 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8ec37e4df4a6a88c09e92952935e3 (DEFAULT_TABLE table=test-table [id=f1ab5420a24f4af1b7aa81005264280b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:58.985481 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a8ec37e4df4a6a88c09e92952935e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:58.985623 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a8ec37e4df4a6a88c09e92952935e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:58.992784 1443 tablet_bootstrap.cc:492] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b: Bootstrap starting.
I20260927 02:43:58.992784 1444 tablet_bootstrap.cc:492] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076: Bootstrap starting.
I20260927 02:43:58.995066 1444 tablet_bootstrap.cc:654] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:58.995066 1443 tablet_bootstrap.cc:654] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:58.995419 1442 tablet_bootstrap.cc:492] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7: Bootstrap starting.
I20260927 02:43:58.995970 1444 log.cc:824] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:58.995965 1443 log.cc:824] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:58.997978 1442 tablet_bootstrap.cc:654] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:58.998083 1443 tablet_bootstrap.cc:492] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b: No bootstrap required, opened a new log
I20260927 02:43:58.998270 1443 ts_tablet_manager.cc:1397] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:43:58.998497 1444 tablet_bootstrap.cc:492] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076: No bootstrap required, opened a new log
I20260927 02:43:58.998695 1444 ts_tablet_manager.cc:1397] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:43:58.999217 1442 log.cc:824] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:59.000824 1442 tablet_bootstrap.cc:492] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7: No bootstrap required, opened a new log
I20260927 02:43:59.001012 1442 ts_tablet_manager.cc:1397] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 02:43:59.002878 1443 raft_consensus.cc:348] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.003000 1444 raft_consensus.cc:348] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.003144 1443 raft_consensus.cc:374] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:59.003197 1443 raft_consensus.cc:729] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77cfedf03c641e791dac383b2f8cd7b, State: Initialized, Role: FOLLOWER
I20260927 02:43:59.003229 1444 raft_consensus.cc:374] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:59.003321 1444 raft_consensus.cc:729] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1dfb0f59fe54de7b8f605e2d1671076, State: Initialized, Role: FOLLOWER
I20260927 02:43:59.003723 1443 consensus_queue.cc:260] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.003723 1444 consensus_queue.cc:260] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.004104 1442 raft_consensus.cc:348] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.004369 1442 raft_consensus.cc:374] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:59.004428 1442 raft_consensus.cc:729] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b5f95d9ef844fdbc53b389bf8fecb7, State: Initialized, Role: FOLLOWER
I20260927 02:43:59.004817 1444 ts_tablet_manager.cc:1428] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:43:59.004817 1443 ts_tablet_manager.cc:1428] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260927 02:43:59.004832 1442 consensus_queue.cc:260] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.005412 1423 heartbeater.cc:503] Master 127.0.98.62:33113 was elected leader, sending a full tablet report...
I20260927 02:43:59.005754 1442 ts_tablet_manager.cc:1428] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:43:59.006028 1288 heartbeater.cc:503] Master 127.0.98.62:33113 was elected leader, sending a full tablet report...
I20260927 02:43:59.025913 1448 raft_consensus.cc:482] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:43:59.026110 1448 raft_consensus.cc:504] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.027226 1448 leader_election.cc:288] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1dfb0f59fe54de7b8f605e2d1671076 (127.0.98.2:33333), d6b5f95d9ef844fdbc53b389bf8fecb7 (127.0.98.3:38111)
I20260927 02:43:59.032893 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:59.027679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49331 (local address 127.0.98.3:38111)
0927 02:43:59.027823 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:59.027827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:59.028098 (+ 271us) server_negotiation.cc:407] Connection header received
0927 02:43:59.028193 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:59.028196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:59.028242 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:59.028330 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:59.028693 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.029500 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.031494 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.032164 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.032293 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:59.032366 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:59.032695 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 02:43:59.032765 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:43:59.032893 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:59.027597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59893 (local address 127.0.98.2:33333)
0927 02:43:59.027742 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:59.027750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:59.028027 (+ 277us) server_negotiation.cc:407] Connection header received
0927 02:43:59.028194 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:59.028197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:59.028242 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:59.028331 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:59.028693 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.029515 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.031576 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.032245 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.032335 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:59.032388 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:59.032698 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 02:43:59.032768 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:43:59.033088 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:59.027484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33333 (local address 127.0.98.1:59893)
0927 02:43:59.027931 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:59.027946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:59.028066 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:59.028395 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:59.028397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:59.028412 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:59.028574 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:59.028598 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.029664 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.029667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:59.031462 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:59.031470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.032347 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.032350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:59.032525 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:59.032562 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:43:59.032675 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:43:59.032802 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":329,"thread_start_us":120,"threads_started":1}
I20260927 02:43:59.033087 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:59.027485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38111 (local address 127.0.98.1:49331)
0927 02:43:59.028010 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:59.028021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:59.028111 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:59.028355 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:59.028361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:59.028385 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:59.028569 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:59.028577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.029664 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.029668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:59.031363 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:59.031375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.032335 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.032340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:59.032520 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:59.032583 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 02:43:59.032668 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:43:59.032808 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":237,"thread_start_us":135,"threads_started":1}
I20260927 02:43:59.033672 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8ec37e4df4a6a88c09e92952935e3" candidate_uuid: "d77cfedf03c641e791dac383b2f8cd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" is_pre_election: true
I20260927 02:43:59.033674 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8ec37e4df4a6a88c09e92952935e3" candidate_uuid: "d77cfedf03c641e791dac383b2f8cd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" is_pre_election: true
I20260927 02:43:59.033951 1242 raft_consensus.cc:2454] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d77cfedf03c641e791dac383b2f8cd7b in term 0.
I20260927 02:43:59.033962 1377 raft_consensus.cc:2454] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d77cfedf03c641e791dac383b2f8cd7b in term 0.
I20260927 02:43:59.034802 1041 leader_election.cc:302] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1dfb0f59fe54de7b8f605e2d1671076, d77cfedf03c641e791dac383b2f8cd7b; no voters:
I20260927 02:43:59.035148 1448 raft_consensus.cc:2781] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:43:59.035250 1448 raft_consensus.cc:482] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:43:59.035324 1448 raft_consensus.cc:3037] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 0 FOLLOWER]: Advancing to term 1
W20260927 02:43:59.076942 1289 tablet.cc:2369] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:43:59.089450 1448 raft_consensus.cc:504] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.089995 1448 leader_election.cc:288] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [CANDIDATE]: Term 1 election: Requested vote from peers d1dfb0f59fe54de7b8f605e2d1671076 (127.0.98.2:33333), d6b5f95d9ef844fdbc53b389bf8fecb7 (127.0.98.3:38111)
I20260927 02:43:59.090389 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8ec37e4df4a6a88c09e92952935e3" candidate_uuid: "d77cfedf03c641e791dac383b2f8cd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1dfb0f59fe54de7b8f605e2d1671076"
I20260927 02:43:59.090560 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8ec37e4df4a6a88c09e92952935e3" candidate_uuid: "d77cfedf03c641e791dac383b2f8cd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7"
I20260927 02:43:59.090602 1242 raft_consensus.cc:3037] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:59.090709 1377 raft_consensus.cc:3037] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:59.092132 1377 raft_consensus.cc:2454] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d77cfedf03c641e791dac383b2f8cd7b in term 1.
I20260927 02:43:59.092195 1242 raft_consensus.cc:2454] T 42a8ec37e4df4a6a88c09e92952935e3 P d1dfb0f59fe54de7b8f605e2d1671076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d77cfedf03c641e791dac383b2f8cd7b in term 1.
I20260927 02:43:59.092530 1043 leader_election.cc:302] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d1dfb0f59fe54de7b8f605e2d1671076, d6b5f95d9ef844fdbc53b389bf8fecb7, d77cfedf03c641e791dac383b2f8cd7b; no voters:
I20260927 02:43:59.092720 1448 raft_consensus.cc:2781] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:59.093006 1448 raft_consensus.cc:686] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [term 1 LEADER]: Becoming Leader. State: Replica: d77cfedf03c641e791dac383b2f8cd7b, State: Running, Role: LEADER
I20260927 02:43:59.093395 1448 consensus_queue.cc:237] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } }
I20260927 02:43:59.097198 962 catalog_manager.cc:5495] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b reported cstate change: term changed from 0 to 1, leader changed from <none> to d77cfedf03c641e791dac383b2f8cd7b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d77cfedf03c641e791dac383b2f8cd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dfb0f59fe54de7b8f605e2d1671076" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d77cfedf03c641e791dac383b2f8cd7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b5f95d9ef844fdbc53b389bf8fecb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38111 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:43:59 dist-test-slave-gkrl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:43:59.136994 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:59.128117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.0.98.1:39317)
0927 02:43:59.128294 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:59.128299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:59.128441 (+ 142us) server_negotiation.cc:407] Connection header received
0927 02:43:59.128601 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:59.128606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:59.128660 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:59.128751 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:59.129440 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.129951 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.130791 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.131007 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.131111 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:59.133728 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:59.133739 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:59.133762 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:59.133784 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:59.135335 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:59.135714 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:59.135719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:59.135720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:59.135791 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:59.136123 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:59.136125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:59.136126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:59.136356 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:59.136503 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:59.136680 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 02:43:59.136810 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 02:43:59 dist-test-slave-gkrl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477039, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 02:43:59.167510 1424 tablet.cc:2369] T 42a8ec37e4df4a6a88c09e92952935e3 P d6b5f95d9ef844fdbc53b389bf8fecb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:43:59 dist-test-slave-gkrl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477039, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:43:59.196527 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:59.186480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.0.98.1:39317)
0927 02:43:59.186651 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:59.186658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:59.186846 (+ 188us) server_negotiation.cc:407] Connection header received
0927 02:43:59.186987 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:59.186993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:59.187057 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:59.187176 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:59.188098 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.189039 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.190046 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.190512 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.190630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:59.193339 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:59.193342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:59.193344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:59.193367 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:59.194912 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:59.195360 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:59.195362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:59.195363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:59.195432 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:59.195726 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:59.195729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:59.195730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:59.195870 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:59.196014 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:59.196319 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 02:43:59.196409 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 02:43:59 dist-test-slave-gkrl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 02:43:59.219640 1154 tablet.cc:2369] T 42a8ec37e4df4a6a88c09e92952935e3 P d77cfedf03c641e791dac383b2f8cd7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:43:59 dist-test-slave-gkrl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:43:59.276209 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:59.267435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.0.98.1:39317)
0927 02:43:59.267602 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:59.267609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:59.267788 (+ 179us) server_negotiation.cc:407] Connection header received
0927 02:43:59.267920 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:59.267924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:59.267986 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:59.268089 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:59.268808 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.269632 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.270501 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:59.270845 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:59.270930 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:59.273294 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:59.273297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:59.273300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:59.273322 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:59.274738 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:59.275129 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:59.275132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:59.275133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:59.275228 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:59.275484 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:59.275486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:59.275487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:59.275642 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:59.275737 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:59.275973 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 02:43:59.276066 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 02:43:59.277915 1067 tablet_service.cc:2908] Rejecting scan request for tablet 42a8ec37e4df4a6a88c09e92952935e3: Uninitialized: clean time has not yet been initialized
I20260927 02:43:59.279915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 1024
I20260927 02:43:59.286393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 1159
I20260927 02:43:59.291836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 1294
I20260927 02:43:59.297036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 930
2026-09-27T02:43:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5213 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:00 dist-test-slave-gkrl krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:00 dist-test-slave-gkrl krb5kdc[1467](info): set up 2 sockets
Sep 27 02:44:00 dist-test-slave-gkrl krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260927 02:44:01.891628 392 mini_kdc.cc:121] Time spent starting KDC: real 2.571s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:01 dist-test-slave-gkrl krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:02.010573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36193
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:02.115701 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:02.115960 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:02.116000 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:02.120236 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:02.120293 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:44:02.120311 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:02.120329 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:02.120345 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:02.120360 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:02.125561 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36193
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:02.126475 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:02.127383 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:02.135509 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:02.135533 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:02.135576 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:02.135604 1480 server_base.cc:1034] running on GCE node
I20260927 02:44:02.136363 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:02.137449 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:02.138622 1480 hybrid_clock.cc:648] HybridClock initialized: now 1790477042138594 us; error 40 us; skew 500 ppm
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:02.141238 1480 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:02.141983 1480 webserver.cc:460] Webserver started at http://127.0.98.62:37889/ using document root <none> and password file <none>
I20260927 02:44:02.142467 1480 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:02.142537 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:02.142699 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:02.144614 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "bda665757d35425ba2206dc0252e5c2f"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "1d6ef74c93295b0a8652254cee098db2"
server_key_iv: "f88d5ffd52ac80cf50e85ed7aafc7306"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.145128 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "bda665757d35425ba2206dc0252e5c2f"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "1d6ef74c93295b0a8652254cee098db2"
server_key_iv: "f88d5ffd52ac80cf50e85ed7aafc7306"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.148739 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:02.150941 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:02.152222 1480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:02.152320 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "bda665757d35425ba2206dc0252e5c2f"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "1d6ef74c93295b0a8652254cee098db2"
server_key_iv: "f88d5ffd52ac80cf50e85ed7aafc7306"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.152397 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:02.173712 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:02.176806 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:02.176944 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:02.184573 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36193
I20260927 02:44:02.184589 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36193 every 8 connection(s)
I20260927 02:44:02.185490 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:02.186133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 1480
I20260927 02:44:02.186244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:02.186507 392 external_mini_cluster.cc:1428] Setting key 3744dd66b9037120ac780f66c423a798
I20260927 02:44:02.188721 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:02.193815 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f: Bootstrap starting.
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:02.195938 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:02.196906 1558 log.cc:824] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:02.198596 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f: No bootstrap required, opened a new log
I20260927 02:44:02.200043 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.187866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.0.98.62:36193)
0927 02:44:02.188305 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.188324 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.188358 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:44:02.189032 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.189052 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.189364 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.189690 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:02.190443 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.191398 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.192311 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.192758 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.192923 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.195347 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:02.195351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:02.195362 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:02.195386 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:02.197195 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.197632 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.197638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.197643 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.197763 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.198043 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.198046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.198048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.198410 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:02.198601 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.198873 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.199157 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":132,"threads_started":1}
I20260927 02:44:02.202093 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda665757d35425ba2206dc0252e5c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260927 02:44:02.202314 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:02.202350 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda665757d35425ba2206dc0252e5c2f, State: Initialized, Role: FOLLOWER
I20260927 02:44:02.202728 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda665757d35425ba2206dc0252e5c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260927 02:44:02.202841 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:02.202885 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:02.202952 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:02.203678 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda665757d35425ba2206dc0252e5c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260927 02:44:02.203997 1558 leader_election.cc:302] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda665757d35425ba2206dc0252e5c2f; no voters:
I20260927 02:44:02.204236 1558 leader_election.cc:288] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:02.204356 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:02.204602 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [term 1 LEADER]: Becoming Leader. State: Replica: bda665757d35425ba2206dc0252e5c2f, State: Running, Role: LEADER
I20260927 02:44:02.204900 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda665757d35425ba2206dc0252e5c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260927 02:44:02.205384 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:02.206441 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda665757d35425ba2206dc0252e5c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda665757d35425ba2206dc0252e5c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } } }
I20260927 02:44:02.206575 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:02.206900 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda665757d35425ba2206dc0252e5c2f. Latest consensus state: current_term: 1 leader_uuid: "bda665757d35425ba2206dc0252e5c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda665757d35425ba2206dc0252e5c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } } }
I20260927 02:44:02.206995 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:02.207254 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:02.210005 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:02.215328 1572 catalog_manager.cc:1347] Generated new cluster ID: 07ef431c3fd14fce849e0c8c641d1df6
I20260927 02:44:02.215387 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:02.230593 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:02.231519 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:02.235124 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f: Generated new TSK 0
I20260927 02:44:02.235730 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:02.262913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:02.370822 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:02.371061 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:02.371104 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:02.375034 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:02.375108 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:02.375168 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:02.375216 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:02.380368 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:02.381244 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:02.382138 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:02.391165 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:02.391165 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:02.391244 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:02.391332 1584 server_base.cc:1034] running on GCE node
I20260927 02:44:02.391755 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:02.392360 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:02.393584 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790477042393571 us; error 44 us; skew 500 ppm
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:02.396358 1584 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:02.397087 1584 webserver.cc:460] Webserver started at http://127.0.98.1:44917/ using document root <none> and password file <none>
I20260927 02:44:02.397567 1584 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:02.397621 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:02.397826 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:02.399714 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "a7dae0d0f869452cb5e0698abe94fd1d"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "4dfc7202bc952310db93398df305410e"
server_key_iv: "deb881ed6159008618df7f7e47f59f88"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.400318 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a7dae0d0f869452cb5e0698abe94fd1d"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "4dfc7202bc952310db93398df305410e"
server_key_iv: "deb881ed6159008618df7f7e47f59f88"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.403746 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:02.406113 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:02.407398 1584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:02.407517 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "a7dae0d0f869452cb5e0698abe94fd1d"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "4dfc7202bc952310db93398df305410e"
server_key_iv: "deb881ed6159008618df7f7e47f59f88"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.407609 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:02.425690 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:02.428880 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:02.429080 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:02.430413 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:02.430460 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:02.430529 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:02.430563 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:02.441107 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33187
I20260927 02:44:02.441185 1714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33187 every 8 connection(s)
I20260927 02:44:02.442148 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:02.444808 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.431282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51403 (local address 127.0.98.62:36193)
0927 02:44:02.431411 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.431415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.432433 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 02:44:02.433601 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.433607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.433675 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.433781 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:02.435293 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.436244 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.437990 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.438359 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.438492 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.441129 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:02.441133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:02.441136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:02.441167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:02.442709 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.443286 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.443292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.443294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.443380 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.443719 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.443722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.443725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.443951 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:02.444070 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.444525 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.444644 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:44:02.445463 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.431787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.1:51403)
0927 02:44:02.432292 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.432330 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.433301 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.433966 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.433976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.434421 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:02.435112 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.435133 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.436376 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.436379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.437851 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.437862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.438511 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.438514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.438647 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.439224 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:02.439246 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:02.440946 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:02.442824 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.442831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.442845 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.443175 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.443478 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.443481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.443483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.443617 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.444066 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:02.444072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:02.444294 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.444467 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.444755 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":142,"threads_started":1}
I20260927 02:44:02.446622 1715 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260927 02:44:02.446822 1715 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:02.447326 1715 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260927 02:44:02.448486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 1584
I20260927 02:44:02.448611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:02.448876 392 external_mini_cluster.cc:1428] Setting key 67d6582896bf093af1b913a7d92f6b24
I20260927 02:44:02.449036 1512 ts_manager.cc:194] Registered new tserver with Master: a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187)
I20260927 02:44:02.450441 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51403
I20260927 02:44:02.457458 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.448937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36307 (local address 127.0.98.62:36193)
0927 02:44:02.449208 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.449212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.449399 (+ 187us) server_negotiation.cc:407] Connection header received
0927 02:44:02.449572 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.449575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.449616 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.449690 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:02.451016 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.451793 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.452713 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.452942 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.453053 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.454231 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:02.454234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:02.454236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:02.454255 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:02.455756 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.456323 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.456326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.456327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.456394 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.456715 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.456717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.456718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.456861 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:02.456929 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.457266 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.457356 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1226,"server-negotiator.queue_time_us":194}
I20260927 02:44:02.457525 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.448901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.1:36307)
0927 02:44:02.449326 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.449340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.449430 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.449776 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.449778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.450097 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:02.450208 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.450214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.451983 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.451988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.452596 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.452604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.453054 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.453057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.453145 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.453637 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:02.453656 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:02.454102 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:02.455862 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.455867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.455892 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.456226 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.456508 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.456513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.456515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.456616 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.456951 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:02.456956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:02.457071 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.457189 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.457317 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:02.476405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:02.583791 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:02.584025 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:02.584067 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:02.587929 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:02.587975 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:02.588012 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:02.588042 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:02.593058 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:02.593923 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:02.594802 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:02.603726 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:02.603746 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:02.603704 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:02.604116 1724 server_base.cc:1034] running on GCE node
I20260927 02:44:02.604470 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:02.605106 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:02.606312 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790477042606286 us; error 51 us; skew 500 ppm
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:02.609103 1724 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:02.609901 1724 webserver.cc:460] Webserver started at http://127.0.98.2:45041/ using document root <none> and password file <none>
I20260927 02:44:02.610440 1724 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:02.610530 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:02.610759 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:02.612871 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a7143fe96874bd8980a5fb3e50f685d"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "0b3c1a055f8d20ded1571edbe9223e92"
server_key_iv: "5876e9c970efcfb75771760a182c91db"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.613421 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a7143fe96874bd8980a5fb3e50f685d"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "0b3c1a055f8d20ded1571edbe9223e92"
server_key_iv: "5876e9c970efcfb75771760a182c91db"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.616945 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:02.619228 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:02.620605 1724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:02.620746 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "8a7143fe96874bd8980a5fb3e50f685d"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "0b3c1a055f8d20ded1571edbe9223e92"
server_key_iv: "5876e9c970efcfb75771760a182c91db"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.620852 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:02.643988 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:02.646713 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:02.646915 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:02.648491 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:02.648541 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:02.648579 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:02.648630 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:02.659072 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41871
I20260927 02:44:02.659240 1854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41871 every 8 connection(s)
I20260927 02:44:02.660019 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:02.661828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 1724
I20260927 02:44:02.661949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:02.662257 392 external_mini_cluster.cc:1428] Setting key 2116302f75a70af4fb7d34f1c30814b8
I20260927 02:44:02.662319 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.649320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50761 (local address 127.0.98.62:36193)
0927 02:44:02.649469 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.649474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.650305 (+ 831us) server_negotiation.cc:407] Connection header received
0927 02:44:02.651417 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.651424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.651513 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.651625 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:02.652888 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.653503 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.654673 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.654890 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.655057 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.658487 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:02.658490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:02.658493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:02.658519 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:02.660290 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.660870 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.660873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.660874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.660951 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.661272 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.661274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.661275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.661438 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:02.661514 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.662072 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.662185 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:44:02.663213 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.649708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.2:50761)
0927 02:44:02.650140 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.650187 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.651132 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.651837 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.651847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.652217 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:02.652732 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.652748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.653645 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.653648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.654520 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.654529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.655110 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.655115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.655355 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.656068 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:02.656110 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:02.658324 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:02.660401 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.660409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.660424 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.660757 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.661045 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.661048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.661050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.661192 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.661526 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:02.661530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:02.661819 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.662028 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.662341 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":130,"threads_started":1}
I20260927 02:44:02.664302 1855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260927 02:44:02.664445 1855 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:02.664885 1855 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260927 02:44:02.665881 1512 ts_manager.cc:194] Registered new tserver with Master: 8a7143fe96874bd8980a5fb3e50f685d (127.0.98.2:41871)
I20260927 02:44:02.666451 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50761
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:02.674052 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.666743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43619 (local address 127.0.98.62:36193)
0927 02:44:02.666842 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.666846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.666964 (+ 118us) server_negotiation.cc:407] Connection header received
0927 02:44:02.667099 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.667102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.667150 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.667225 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:02.667779 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.668520 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.669392 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.669624 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.669731 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.670776 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:02.670780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:02.670783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:02.670805 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:02.672331 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.672761 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.672775 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.672777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.672863 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.673212 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.673216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.673218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.673416 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:02.673523 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.673779 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.673895 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:44:02.674078 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.666545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.2:43619)
0927 02:44:02.666871 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.666888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.667005 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.667266 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.667268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.667493 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:02.667629 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.667639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.668664 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.668668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.669272 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.669281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.669731 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.669734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.669821 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.670319 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:02.670329 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:02.670647 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:02.672443 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.672448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.672450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.672658 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.673008 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.673012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.673022 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.673112 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.673513 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:02.673516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:02.673617 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.673727 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.673897 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:02.691188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:02.799854 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:02.800221 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:02.800282 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:02.804229 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:02.804276 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:02.804314 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:02.804345 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:02.809508 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:02.810515 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:02.811484 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:44:02.820933 1864 server_base.cc:1034] running on GCE node
W20260927 02:44:02.820994 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:02.820899 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:02.820899 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:02.821491 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:02.822028 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:02.823184 1864 hybrid_clock.cc:648] HybridClock initialized: now 1790477042823163 us; error 34 us; skew 500 ppm
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:02.826084 1864 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:02.826889 1864 webserver.cc:460] Webserver started at http://127.0.98.3:36197/ using document root <none> and password file <none>
I20260927 02:44:02.827365 1864 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:02.827420 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:02.827567 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:02.829521 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "32f2833a7f1a4d898752bb8ae629d64c"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "4abc3841bf0a630c7c0eef77b50f3fe4"
server_key_iv: "f132a0e149f13de0f2b3c3b7ec24c2b0"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.830017 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "32f2833a7f1a4d898752bb8ae629d64c"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "4abc3841bf0a630c7c0eef77b50f3fe4"
server_key_iv: "f132a0e149f13de0f2b3c3b7ec24c2b0"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.833460 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:02.835601 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:02.836834 1864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:02.836971 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "32f2833a7f1a4d898752bb8ae629d64c"
format_stamp: "Formatted at 2026-09-27 02:44:02 on dist-test-slave-gkrl"
server_key: "4abc3841bf0a630c7c0eef77b50f3fe4"
server_key_iv: "f132a0e149f13de0f2b3c3b7ec24c2b0"
server_key_version: "encryptionkey@0"
I20260927 02:44:02.837078 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:02.850238 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:02.853371 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:02.853567 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:02.854947 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:02.855017 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:02.855077 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:02.855130 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:02.865832 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44683
I20260927 02:44:02.865939 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44683 every 8 connection(s)
I20260927 02:44:02.866775 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:02.868736 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.855709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35797 (local address 127.0.98.62:36193)
0927 02:44:02.855832 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.855836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.856928 (+ 1092us) server_negotiation.cc:407] Connection header received
0927 02:44:02.857970 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.857974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.858040 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.858119 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:02.859603 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.860529 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.861899 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.862256 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.862374 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.865160 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:02.865164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:02.865167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:02.865213 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:02.866772 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.867336 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.867339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.867341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.867421 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.867789 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.867792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.867793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.868027 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:02.868112 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.868524 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.868623 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:44:02.869544 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.856330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.3:35797)
0927 02:44:02.856778 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.856819 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.857724 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.858557 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.858565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.858899 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:02.859421 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.859438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.860671 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.860675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.861765 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.861774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.862414 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.862418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.862581 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.863144 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:02.863175 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:02.864993 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:02.866893 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.866899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.866912 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.867220 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.867518 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.867521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.867523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.867683 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.868117 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:02.868121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:02.868327 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.868476 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.868707 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
I20260927 02:44:02.870589 1995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260927 02:44:02.870726 1995 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:02.871227 1995 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260927 02:44:02.872287 1510 ts_manager.cc:194] Registered new tserver with Master: 32f2833a7f1a4d898752bb8ae629d64c (127.0.98.3:44683)
I20260927 02:44:02.873065 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35797
I20260927 02:44:02.876767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 1864
I20260927 02:44:02.876879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:02.877170 392 external_mini_cluster.cc:1428] Setting key 6096126b952049265624c55d9f2515ce
I20260927 02:44:02.879347 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:02.879796 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.873008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44065 (local address 127.0.98.62:36193)
0927 02:44:02.873130 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.873134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.873150 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:02.873245 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.873248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.873286 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.873364 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:02.873883 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.874349 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.875373 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.875566 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.875650 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.876607 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:02.876610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:02.876612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:02.876631 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:02.878108 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.878636 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.878640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.878642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.878721 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.879008 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.879014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.879016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.879188 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:02.879297 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.879543 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.879647 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:44:02.879812 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.872749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.3:44065)
0927 02:44:02.873036 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.873051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.873142 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.873400 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.873403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.873643 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:02.873755 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.873762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.874494 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.874497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.875271 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.875279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.875671 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.875674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.875739 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.876172 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:02.876183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:02.876472 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:02.878247 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.878252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.878255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.878533 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.878829 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:02.878832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:02.878834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:02.878912 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:02.879289 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:02.879292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:02.879385 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.879492 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.879629 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":94,"threads_started":1}
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:44:02 dist-test-slave-gkrl krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:02.903838 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.895795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.0.98.62:36193)
0927 02:44:02.895963 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.895967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.895978 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:44:02.896056 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.896059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.896097 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.896192 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:02.896868 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.897373 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.898204 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.898423 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.898534 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.900830 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:02.900835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:02.900843 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:02.900864 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:02.902216 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.902658 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.902663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.902665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.902742 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:02.903011 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:02.903015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:02.903017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:02.903190 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:02.903318 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.903562 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.903670 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:44:02.906756 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:02.909858 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:02.923566 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.917755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41871 (local address 127.0.0.1:45236)
0927 02:44:02.918352 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.918400 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.918532 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.919310 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.919316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.919338 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:02.919453 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.919459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.920757 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.920763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.922064 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.922073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.923056 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.923059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.923163 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.923196 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.923313 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.923429 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":64,"threads_started":1}
I20260927 02:44:02.923578 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.917890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.0.98.2:41871)
0927 02:44:02.918277 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.918289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.918685 (+ 396us) server_negotiation.cc:407] Connection header received
0927 02:44:02.918770 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.918777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.918988 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.919148 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:02.919572 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.920643 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.922191 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.922937 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.923057 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.923232 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.923342 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.923429 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":109,"threads_started":1}
I20260927 02:44:02.923851 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.917854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44683 (local address 127.0.0.1:50522)
0927 02:44:02.918754 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.918770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.918853 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.919668 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.919671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.919688 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:02.919793 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.919806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.921105 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.921108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.922266 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.922273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.923399 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.923403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.923502 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.923533 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.923600 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.923669 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":26,"thread_start_us":90,"threads_started":1}
I20260927 02:44:02.924299 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.918412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.0.98.3:44683)
0927 02:44:02.919238 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.919255 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.919278 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:44:02.919341 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.919345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.919518 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.919684 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:02.919921 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.920989 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.922382 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.923071 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.923188 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.923334 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.924036 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.924154 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":104,"threads_started":1}
I20260927 02:44:02.924573 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.918451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33187 (local address 127.0.0.1:42936)
0927 02:44:02.919175 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:02.919188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:02.919273 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:02.919970 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:02.919973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:02.919990 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:02.920082 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.920088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.921381 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.921385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.922572 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:02.922584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.924179 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.924182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:02.924282 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.924308 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:44:02.924380 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:44:02.924452 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":87,"thread_start_us":79,"threads_started":1}
I20260927 02:44:02.925335 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:02.918659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.0.98.1:33187)
0927 02:44:02.919007 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:02.919040 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:02.919392 (+ 352us) server_negotiation.cc:407] Connection header received
0927 02:44:02.919470 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:02.919474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:02.919670 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:02.919812 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:02.920209 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.921241 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.922720 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:02.923494 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:02.923602 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:02.924985 (+ 1383us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:02.925061 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 02:44:02.925208 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":82,"threads_started":1}
I20260927 02:44:02.927395 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 88c62849cb9f4c7b9effe98139f71907 (DEFAULT_TABLE table=test-table [id=f8de7b2bc6394b6b997781eb877f2896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:02.927320 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 88c62849cb9f4c7b9effe98139f71907 (DEFAULT_TABLE table=test-table [id=f8de7b2bc6394b6b997781eb877f2896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:02.927320 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 88c62849cb9f4c7b9effe98139f71907 (DEFAULT_TABLE table=test-table [id=f8de7b2bc6394b6b997781eb877f2896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:02.928364 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c62849cb9f4c7b9effe98139f71907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:02.928371 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c62849cb9f4c7b9effe98139f71907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:02.928386 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c62849cb9f4c7b9effe98139f71907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:02.934911 2018 tablet_bootstrap.cc:492] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Bootstrap starting.
I20260927 02:44:02.936165 2019 tablet_bootstrap.cc:492] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Bootstrap starting.
I20260927 02:44:02.936484 2020 tablet_bootstrap.cc:492] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Bootstrap starting.
I20260927 02:44:02.937347 2018 tablet_bootstrap.cc:654] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:02.938052 2018 log.cc:824] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:02.938174 2019 tablet_bootstrap.cc:654] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:02.938189 2020 tablet_bootstrap.cc:654] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:02.938987 2019 log.cc:824] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:02.938987 2020 log.cc:824] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:02.940301 2018 tablet_bootstrap.cc:492] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: No bootstrap required, opened a new log
I20260927 02:44:02.940497 2018 ts_tablet_manager.cc:1397] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:44:02.940831 2019 tablet_bootstrap.cc:492] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: No bootstrap required, opened a new log
I20260927 02:44:02.941006 2019 ts_tablet_manager.cc:1397] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:44:02.941154 2020 tablet_bootstrap.cc:492] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: No bootstrap required, opened a new log
I20260927 02:44:02.941324 2020 ts_tablet_manager.cc:1397] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:44:02.943579 2018 raft_consensus.cc:348] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:02.943825 2018 raft_consensus.cc:374] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:02.943877 2018 raft_consensus.cc:729] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7dae0d0f869452cb5e0698abe94fd1d, State: Initialized, Role: FOLLOWER
I20260927 02:44:02.944329 2018 consensus_queue.cc:260] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:02.945138 2018 ts_tablet_manager.cc:1428] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 02:44:02.945233 2019 raft_consensus.cc:348] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:02.945310 2020 raft_consensus.cc:348] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:02.945475 2019 raft_consensus.cc:374] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:02.945490 2020 raft_consensus.cc:374] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:02.945511 1715 heartbeater.cc:503] Master 127.0.98.62:36193 was elected leader, sending a full tablet report...
I20260927 02:44:02.945533 2020 raft_consensus.cc:729] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7143fe96874bd8980a5fb3e50f685d, State: Initialized, Role: FOLLOWER
I20260927 02:44:02.945530 2019 raft_consensus.cc:729] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f2833a7f1a4d898752bb8ae629d64c, State: Initialized, Role: FOLLOWER
I20260927 02:44:02.945925 2019 consensus_queue.cc:260] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:02.945925 2020 consensus_queue.cc:260] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:02.946885 1995 heartbeater.cc:503] Master 127.0.98.62:36193 was elected leader, sending a full tablet report...
I20260927 02:44:02.946933 2020 ts_tablet_manager.cc:1428] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:44:02.947212 1855 heartbeater.cc:503] Master 127.0.98.62:36193 was elected leader, sending a full tablet report...
I20260927 02:44:02.946869 2019 ts_tablet_manager.cc:1428] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 02:44:03.117731 1996 tablet.cc:2369] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:44:03.161509 1856 tablet.cc:2369] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:44:03.193425 1716 tablet.cc:2369] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:03.242962 2025 raft_consensus.cc:482] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:03.243176 2025 raft_consensus.cc:504] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:03.244308 2025 leader_election.cc:288] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a7143fe96874bd8980a5fb3e50f685d (127.0.98.2:41871), a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187)
I20260927 02:44:03.248484 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.244628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41871 (local address 127.0.98.3:44863)
0927 02:44:03.244774 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:03.244792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:03.244896 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:03.245276 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:03.245279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:03.245302 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:03.245433 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.245439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.246265 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.246268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.247365 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.247379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.248094 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.248098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.248214 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.248244 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 02:44:03.248317 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:44:03.248376 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":49}
I20260927 02:44:03.248549 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.244699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44863 (local address 127.0.98.2:41871)
0927 02:44:03.244939 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.245005 (+ 66us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.245025 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:44:03.245080 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.245082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.245149 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.245248 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:03.245565 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.246159 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.247514 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.247989 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.248075 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.248127 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.248348 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.248428 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260927 02:44:03.249186 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c62849cb9f4c7b9effe98139f71907" candidate_uuid: "32f2833a7f1a4d898752bb8ae629d64c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a7143fe96874bd8980a5fb3e50f685d" is_pre_election: true
I20260927 02:44:03.249395 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.244938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33187 (local address 127.0.98.3:56857)
0927 02:44:03.245333 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:03.245347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:03.245440 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:03.245718 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:03.245720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:03.245733 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:03.246165 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.246172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.247007 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.247011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.248153 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.248161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.248996 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.249000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.249087 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.249109 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:44:03.249193 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:44:03.249258 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":9,"thread_start_us":106,"threads_started":1}
I20260927 02:44:03.249402 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.245075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56857 (local address 127.0.98.1:33187)
0927 02:44:03.245231 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.245238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.245454 (+ 216us) server_negotiation.cc:407] Connection header received
0927 02:44:03.245550 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.245553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.245611 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.245701 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:03.246317 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.246900 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.248278 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.248851 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.248953 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.249000 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.249212 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.249266 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:44:03.249521 1809 raft_consensus.cc:2454] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32f2833a7f1a4d898752bb8ae629d64c in term 0.
I20260927 02:44:03.250046 1883 leader_election.cc:302] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f2833a7f1a4d898752bb8ae629d64c, 8a7143fe96874bd8980a5fb3e50f685d; no voters:
I20260927 02:44:03.250041 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c62849cb9f4c7b9effe98139f71907" candidate_uuid: "32f2833a7f1a4d898752bb8ae629d64c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" is_pre_election: true
I20260927 02:44:03.250363 1669 raft_consensus.cc:2454] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32f2833a7f1a4d898752bb8ae629d64c in term 0.
I20260927 02:44:03.250355 2025 raft_consensus.cc:2781] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:03.250445 2025 raft_consensus.cc:482] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:03.250488 2025 raft_consensus.cc:3037] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.251520 2025 raft_consensus.cc:504] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:03.251809 2025 leader_election.cc:288] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [CANDIDATE]: Term 1 election: Requested vote from peers 8a7143fe96874bd8980a5fb3e50f685d (127.0.98.2:41871), a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187)
I20260927 02:44:03.252157 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c62849cb9f4c7b9effe98139f71907" candidate_uuid: "32f2833a7f1a4d898752bb8ae629d64c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a7143fe96874bd8980a5fb3e50f685d"
I20260927 02:44:03.252226 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c62849cb9f4c7b9effe98139f71907" candidate_uuid: "32f2833a7f1a4d898752bb8ae629d64c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dae0d0f869452cb5e0698abe94fd1d"
I20260927 02:44:03.252277 1809 raft_consensus.cc:3037] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.252358 1669 raft_consensus.cc:3037] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.253257 1809 raft_consensus.cc:2454] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32f2833a7f1a4d898752bb8ae629d64c in term 1.
I20260927 02:44:03.253383 1669 raft_consensus.cc:2454] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32f2833a7f1a4d898752bb8ae629d64c in term 1.
I20260927 02:44:03.253556 1883 leader_election.cc:302] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f2833a7f1a4d898752bb8ae629d64c, 8a7143fe96874bd8980a5fb3e50f685d; no voters:
I20260927 02:44:03.253748 2025 raft_consensus.cc:2781] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:03.253933 2025 raft_consensus.cc:686] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 LEADER]: Becoming Leader. State: Replica: 32f2833a7f1a4d898752bb8ae629d64c, State: Running, Role: LEADER
I20260927 02:44:03.254264 2025 consensus_queue.cc:237] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } }
I20260927 02:44:03.257519 1510 catalog_manager.cc:5495] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c reported cstate change: term changed from 0 to 1, leader changed from <none> to 32f2833a7f1a4d898752bb8ae629d64c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "32f2833a7f1a4d898752bb8ae629d64c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:03.289383 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.285999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.0.98.3:44683)
0927 02:44:03.286131 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.286135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.286148 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:03.286255 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.286257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.286308 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.286388 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:03.286624 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.287121 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.288020 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.288252 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.288346 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.288641 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:03.289038 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:03.289135 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.289209 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.289262 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:44:03.294936 1669 raft_consensus.cc:1261] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Refusing update from remote peer 32f2833a7f1a4d898752bb8ae629d64c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:03.295032 1809 raft_consensus.cc:1261] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Refusing update from remote peer 32f2833a7f1a4d898752bb8ae629d64c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:03.295653 2025 consensus_queue.cc:1035] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:03.295955 2028 consensus_queue.cc:1035] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:03.303381 2033 mvcc.cc:205] Tried to move back new op lower bound from 7333793969328734208 to 7333793969171910656. Current Snapshot: MvccSnapshot[applied={T|T < 7333793969328734208}]
I20260927 02:44:03.313287 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:03.313455 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:03.315721 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f: Sending DeleteTablet for 3 replicas of tablet 88c62849cb9f4c7b9effe98139f71907
I20260927 02:44:03.316567 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c62849cb9f4c7b9effe98139f71907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45236
I20260927 02:44:03.316799 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c62849cb9f4c7b9effe98139f71907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42936
I20260927 02:44:03.316995 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c62849cb9f4c7b9effe98139f71907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50522
I20260927 02:44:03.317400 2043 tablet_replica.cc:330] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: stopping tablet replica
I20260927 02:44:03.317401 2041 tablet_replica.cc:330] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: stopping tablet replica
I20260927 02:44:03.317498 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:44:03.317703 2041 raft_consensus.cc:2229] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 02:44:03.317886 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:03.317982 2041 raft_consensus.cc:2258] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:03.317935 2043 raft_consensus.cc:2229] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:03.318742 2043 raft_consensus.cc:2258] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:03.319793 2041 ts_tablet_manager.cc:1905] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:03.322338 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 88541950438b42a99e8d8f83be524796 (DEFAULT_TABLE table=test-table [id=64a07492881b4c4fafb5b7ba22c5d954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:03.322611 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88541950438b42a99e8d8f83be524796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:03.322566 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 88541950438b42a99e8d8f83be524796 (DEFAULT_TABLE table=test-table [id=64a07492881b4c4fafb5b7ba22c5d954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:03.322552 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 88541950438b42a99e8d8f83be524796 (DEFAULT_TABLE table=test-table [id=64a07492881b4c4fafb5b7ba22c5d954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:03.322783 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88541950438b42a99e8d8f83be524796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:03.322803 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88541950438b42a99e8d8f83be524796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:03.323858 2042 tablet_replica.cc:330] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: stopping tablet replica
I20260927 02:44:03.324178 2042 raft_consensus.cc:2229] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:03.324424 2042 raft_consensus.cc:2258] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:03.325037 2019 tablet_bootstrap.cc:492] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: Bootstrap starting.
I20260927 02:44:03.325052 2018 tablet_bootstrap.cc:492] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: Bootstrap starting.
I20260927 02:44:03.325129 2020 tablet_bootstrap.cc:492] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: Bootstrap starting.
I20260927 02:44:03.325750 2018 tablet_bootstrap.cc:654] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:03.325953 2019 tablet_bootstrap.cc:654] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:03.325953 2020 tablet_bootstrap.cc:654] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:03.326768 2018 tablet_bootstrap.cc:492] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: No bootstrap required, opened a new log
I20260927 02:44:03.326848 2018 ts_tablet_manager.cc:1397] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:03.327453 2018 raft_consensus.cc:348] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.327564 2018 raft_consensus.cc:374] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:03.327603 2018 raft_consensus.cc:729] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7dae0d0f869452cb5e0698abe94fd1d, State: Initialized, Role: FOLLOWER
I20260927 02:44:03.327721 2018 consensus_queue.cc:260] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.327718 2042 ts_tablet_manager.cc:1905] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:03.328027 2018 ts_tablet_manager.cc:1428] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 02:44:03.330327 2042 ts_tablet_manager.cc:1918] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:03.330420 2042 log.cc:1196] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/88c62849cb9f4c7b9effe98139f71907
I20260927 02:44:03.330730 2042 ts_tablet_manager.cc:1939] T 88c62849cb9f4c7b9effe98139f71907 P a7dae0d0f869452cb5e0698abe94fd1d: Deleting consensus metadata
I20260927 02:44:03.331804 1499 catalog_manager.cc:4841] TS a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187): tablet 88c62849cb9f4c7b9effe98139f71907 (table test-table [id=f8de7b2bc6394b6b997781eb877f2896]) successfully deleted
I20260927 02:44:03.332674 2043 ts_tablet_manager.cc:1905] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:03.332726 2020 tablet_bootstrap.cc:492] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: No bootstrap required, opened a new log
I20260927 02:44:03.332800 2020 ts_tablet_manager.cc:1397] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260927 02:44:03.332935 2019 tablet_bootstrap.cc:492] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: No bootstrap required, opened a new log
I20260927 02:44:03.333005 2019 ts_tablet_manager.cc:1397] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 02:44:03.333473 2019 raft_consensus.cc:348] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.333570 2019 raft_consensus.cc:374] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:03.333606 2019 raft_consensus.cc:729] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f2833a7f1a4d898752bb8ae629d64c, State: Initialized, Role: FOLLOWER
I20260927 02:44:03.333604 2020 raft_consensus.cc:348] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.333701 2020 raft_consensus.cc:374] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:03.333734 2020 raft_consensus.cc:729] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7143fe96874bd8980a5fb3e50f685d, State: Initialized, Role: FOLLOWER
I20260927 02:44:03.333721 2019 consensus_queue.cc:260] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.333842 2020 consensus_queue.cc:260] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.333966 2019 ts_tablet_manager.cc:1428] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:03.334084 2020 ts_tablet_manager.cc:1428] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:03.334939 2043 ts_tablet_manager.cc:1918] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:03.335019 2043 log.cc:1196] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/88c62849cb9f4c7b9effe98139f71907
I20260927 02:44:03.335291 2043 ts_tablet_manager.cc:1939] T 88c62849cb9f4c7b9effe98139f71907 P 32f2833a7f1a4d898752bb8ae629d64c: Deleting consensus metadata
I20260927 02:44:03.336308 2041 ts_tablet_manager.cc:1918] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:03.336398 2041 log.cc:1196] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/88c62849cb9f4c7b9effe98139f71907
I20260927 02:44:03.336692 2041 ts_tablet_manager.cc:1939] T 88c62849cb9f4c7b9effe98139f71907 P 8a7143fe96874bd8980a5fb3e50f685d: Deleting consensus metadata
I20260927 02:44:03.337678 1499 catalog_manager.cc:4841] TS 8a7143fe96874bd8980a5fb3e50f685d (127.0.98.2:41871): tablet 88c62849cb9f4c7b9effe98139f71907 (table test-table [id=f8de7b2bc6394b6b997781eb877f2896]) successfully deleted
I20260927 02:44:03.337951 1497 catalog_manager.cc:4841] TS 32f2833a7f1a4d898752bb8ae629d64c (127.0.98.3:44683): tablet 88c62849cb9f4c7b9effe98139f71907 (table test-table [id=f8de7b2bc6394b6b997781eb877f2896]) successfully deleted
I20260927 02:44:03.342243 2036 raft_consensus.cc:482] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:03.342343 2036 raft_consensus.cc:504] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.342746 2036 leader_election.cc:288] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187), 8a7143fe96874bd8980a5fb3e50f685d (127.0.98.2:41871)
I20260927 02:44:03.343108 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88541950438b42a99e8d8f83be524796" candidate_uuid: "32f2833a7f1a4d898752bb8ae629d64c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" is_pre_election: true
I20260927 02:44:03.343180 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88541950438b42a99e8d8f83be524796" candidate_uuid: "32f2833a7f1a4d898752bb8ae629d64c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a7143fe96874bd8980a5fb3e50f685d" is_pre_election: true
I20260927 02:44:03.343236 1669 raft_consensus.cc:2454] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32f2833a7f1a4d898752bb8ae629d64c in term 0.
I20260927 02:44:03.343302 1809 raft_consensus.cc:2454] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32f2833a7f1a4d898752bb8ae629d64c in term 0.
I20260927 02:44:03.343523 1883 leader_election.cc:302] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f2833a7f1a4d898752bb8ae629d64c, a7dae0d0f869452cb5e0698abe94fd1d; no voters:
I20260927 02:44:03.343683 2036 raft_consensus.cc:2781] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:03.343740 2036 raft_consensus.cc:482] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:03.343780 2036 raft_consensus.cc:3037] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.344589 2036 raft_consensus.cc:504] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.344919 2036 leader_election.cc:288] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [CANDIDATE]: Term 1 election: Requested vote from peers a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187), 8a7143fe96874bd8980a5fb3e50f685d (127.0.98.2:41871)
I20260927 02:44:03.345211 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88541950438b42a99e8d8f83be524796" candidate_uuid: "32f2833a7f1a4d898752bb8ae629d64c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dae0d0f869452cb5e0698abe94fd1d"
I20260927 02:44:03.345253 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88541950438b42a99e8d8f83be524796" candidate_uuid: "32f2833a7f1a4d898752bb8ae629d64c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a7143fe96874bd8980a5fb3e50f685d"
I20260927 02:44:03.345346 1809 raft_consensus.cc:3037] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.345341 1669 raft_consensus.cc:3037] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.346127 1669 raft_consensus.cc:2454] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32f2833a7f1a4d898752bb8ae629d64c in term 1.
I20260927 02:44:03.346341 1809 raft_consensus.cc:2454] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32f2833a7f1a4d898752bb8ae629d64c in term 1.
I20260927 02:44:03.346376 1883 leader_election.cc:302] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f2833a7f1a4d898752bb8ae629d64c, a7dae0d0f869452cb5e0698abe94fd1d; no voters:
I20260927 02:44:03.346522 2036 raft_consensus.cc:2781] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:03.346642 2036 raft_consensus.cc:686] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 LEADER]: Becoming Leader. State: Replica: 32f2833a7f1a4d898752bb8ae629d64c, State: Running, Role: LEADER
I20260927 02:44:03.346810 2036 consensus_queue.cc:237] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.348263 1511 catalog_manager.cc:5495] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c reported cstate change: term changed from 0 to 1, leader changed from <none> to 32f2833a7f1a4d898752bb8ae629d64c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "32f2833a7f1a4d898752bb8ae629d64c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } health_report { overall_health: HEALTHY } } }
Sep 27 02:44:03 dist-test-slave-gkrl krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:03.391402 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.377540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.0.1:39250)
0927 02:44:03.377818 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:03.377833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:03.377939 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:03.378214 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:03.378217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:03.378816 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:03.378966 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.378974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.379722 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.379726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.380656 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.380666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.381107 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.381111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.381257 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.386411 (+ 5154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:03.386427 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:03.388146 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:03.389955 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:03.389963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:03.389968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:03.390197 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:03.390478 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:03.390480 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:03.390481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:03.390555 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:03.390936 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:03.390941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:03.391076 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 02:44:03.391169 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:44:03.391273 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":40,"thread_start_us":98,"threads_started":1}
I20260927 02:44:03.391532 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.377613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.0.98.62:36193)
0927 02:44:03.377737 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.377741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.377939 (+ 198us) server_negotiation.cc:407] Connection header received
0927 02:44:03.378057 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.378060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.378098 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.378171 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:03.379107 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.379622 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.380771 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.380982 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.381101 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.388273 (+ 7172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:03.388277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:03.388280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:03.388304 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:03.389854 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:03.390306 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:03.390310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:03.390312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:03.390391 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:03.390638 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:03.390642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:03.390644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:03.390842 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:03.390934 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.391242 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.391352 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":52}
I20260927 02:44:03.395411 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:44:03.396238 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:03.400795 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 52fa8c3336934a5788a4d5818d5f3ded (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fcbe6375266d4fa794c2e884090638a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:03.400956 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 52fa8c3336934a5788a4d5818d5f3ded (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fcbe6375266d4fa794c2e884090638a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:03.401171 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fa8c3336934a5788a4d5818d5f3ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:03.401289 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fa8c3336934a5788a4d5818d5f3ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:03.403437 2018 tablet_bootstrap.cc:492] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: Bootstrap starting.
I20260927 02:44:03.403792 2019 tablet_bootstrap.cc:492] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: Bootstrap starting.
I20260927 02:44:03.403800 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 52fa8c3336934a5788a4d5818d5f3ded (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fcbe6375266d4fa794c2e884090638a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:03.404183 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fa8c3336934a5788a4d5818d5f3ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:03.404404 2018 tablet_bootstrap.cc:654] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:03.404657 2019 tablet_bootstrap.cc:654] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:03.405745 2019 tablet_bootstrap.cc:492] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: No bootstrap required, opened a new log
I20260927 02:44:03.405821 2019 ts_tablet_manager.cc:1397] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:03.406207 2019 raft_consensus.cc:348] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.406313 2019 raft_consensus.cc:374] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:03.406342 2019 raft_consensus.cc:729] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f2833a7f1a4d898752bb8ae629d64c, State: Initialized, Role: FOLLOWER
I20260927 02:44:03.406450 2019 consensus_queue.cc:260] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.406687 2019 ts_tablet_manager.cc:1428] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:03.406867 2036 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.406978 2036 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:03.407336 2020 tablet_bootstrap.cc:492] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: Bootstrap starting.
I20260927 02:44:03.408782 2020 tablet_bootstrap.cc:654] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:03.409685 2018 tablet_bootstrap.cc:492] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: No bootstrap required, opened a new log
I20260927 02:44:03.409780 2018 ts_tablet_manager.cc:1397] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 02:44:03.410008 2020 tablet_bootstrap.cc:492] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: No bootstrap required, opened a new log
I20260927 02:44:03.410089 2020 ts_tablet_manager.cc:1397] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 02:44:03.410233 2018 raft_consensus.cc:348] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.410341 2018 raft_consensus.cc:374] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:03.410379 2018 raft_consensus.cc:729] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7dae0d0f869452cb5e0698abe94fd1d, State: Initialized, Role: FOLLOWER
I20260927 02:44:03.410506 2018 consensus_queue.cc:260] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.410566 2020 raft_consensus.cc:348] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.410676 2020 raft_consensus.cc:374] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:03.410794 2018 ts_tablet_manager.cc:1428] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:03.410787 2020 raft_consensus.cc:729] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7143fe96874bd8980a5fb3e50f685d, State: Initialized, Role: FOLLOWER
I20260927 02:44:03.411005 2024 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.411159 2024 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:03.410943 2020 consensus_queue.cc:260] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.411619 2020 ts_tablet_manager.cc:1428] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:03.411550 2026 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.412269 2026 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:03.553682 2026 raft_consensus.cc:482] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:03.553819 2026 raft_consensus.cc:504] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.554982 2026 leader_election.cc:288] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187), 32f2833a7f1a4d898752bb8ae629d64c (127.0.98.3:44683)
I20260927 02:44:03.559535 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.555385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34813 (local address 127.0.98.1:33187)
0927 02:44:03.555515 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.555520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.555974 (+ 454us) server_negotiation.cc:407] Connection header received
0927 02:44:03.556149 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.556152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.556227 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.556306 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:03.556639 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.557314 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.558545 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.558916 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.559000 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.559086 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.559310 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.559391 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:44:03.559582 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.555379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35049 (local address 127.0.98.3:44683)
0927 02:44:03.555551 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.555556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.555803 (+ 247us) server_negotiation.cc:407] Connection header received
0927 02:44:03.555979 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.555983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.556040 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.556123 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:03.556748 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.557443 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.558519 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.558980 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.559105 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.559174 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.559362 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.559440 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 02:44:03.559623 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.555323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44683 (local address 127.0.98.2:35049)
0927 02:44:03.555683 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:03.555695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:03.555844 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:03.556161 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:03.556164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:03.556197 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:03.556598 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.556609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.557580 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.557586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.558391 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.558399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.559119 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.559123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.559216 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.559232 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:44:03.559329 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:44:03.559412 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":222,"thread_start_us":115,"threads_started":1}
I20260927 02:44:03.560061 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.555326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33187 (local address 127.0.98.2:34813)
0927 02:44:03.555833 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:03.555847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:03.556008 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:03.556333 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:03.556336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:03.556350 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:03.556504 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.556514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.557441 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.557444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.558400 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.558410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.559042 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.559049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.559164 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.559196 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:44:03.559280 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:44:03.559412 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":96,"thread_start_us":120,"threads_started":1}
I20260927 02:44:03.560139 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fa8c3336934a5788a4d5818d5f3ded" candidate_uuid: "8a7143fe96874bd8980a5fb3e50f685d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32f2833a7f1a4d898752bb8ae629d64c" is_pre_election: true
I20260927 02:44:03.560402 1949 raft_consensus.cc:2454] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a7143fe96874bd8980a5fb3e50f685d in term 0.
I20260927 02:44:03.560616 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fa8c3336934a5788a4d5818d5f3ded" candidate_uuid: "8a7143fe96874bd8980a5fb3e50f685d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" is_pre_election: true
I20260927 02:44:03.560778 1669 raft_consensus.cc:2454] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a7143fe96874bd8980a5fb3e50f685d in term 0.
I20260927 02:44:03.560869 1741 leader_election.cc:302] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f2833a7f1a4d898752bb8ae629d64c, 8a7143fe96874bd8980a5fb3e50f685d; no voters:
I20260927 02:44:03.561158 2026 raft_consensus.cc:2781] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:03.561219 2026 raft_consensus.cc:482] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:03.561257 2026 raft_consensus.cc:3037] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.562197 2026 raft_consensus.cc:504] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.562515 2026 leader_election.cc:288] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [CANDIDATE]: Term 1 election: Requested vote from peers a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187), 32f2833a7f1a4d898752bb8ae629d64c (127.0.98.3:44683)
I20260927 02:44:03.562834 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fa8c3336934a5788a4d5818d5f3ded" candidate_uuid: "8a7143fe96874bd8980a5fb3e50f685d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32f2833a7f1a4d898752bb8ae629d64c"
I20260927 02:44:03.562829 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fa8c3336934a5788a4d5818d5f3ded" candidate_uuid: "8a7143fe96874bd8980a5fb3e50f685d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dae0d0f869452cb5e0698abe94fd1d"
I20260927 02:44:03.562947 1669 raft_consensus.cc:3037] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.562945 1949 raft_consensus.cc:3037] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:03.564052 1949 raft_consensus.cc:2454] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a7143fe96874bd8980a5fb3e50f685d in term 1.
I20260927 02:44:03.564039 1669 raft_consensus.cc:2454] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a7143fe96874bd8980a5fb3e50f685d in term 1.
I20260927 02:44:03.564316 1741 leader_election.cc:302] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f2833a7f1a4d898752bb8ae629d64c, 8a7143fe96874bd8980a5fb3e50f685d; no voters:
I20260927 02:44:03.564528 2026 raft_consensus.cc:2781] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:03.564741 2026 raft_consensus.cc:686] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [term 1 LEADER]: Becoming Leader. State: Replica: 8a7143fe96874bd8980a5fb3e50f685d, State: Running, Role: LEADER
I20260927 02:44:03.565006 2026 consensus_queue.cc:237] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } }
I20260927 02:44:03.566628 2059 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: TxnStatusTablet state changed. Reason: New leader 8a7143fe96874bd8980a5fb3e50f685d. Latest consensus state: current_term: 1 leader_uuid: "8a7143fe96874bd8980a5fb3e50f685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.566738 2059 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:03.567030 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:44:03.567193 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:44:03.567582 1511 catalog_manager.cc:5495] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a7143fe96874bd8980a5fb3e50f685d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8a7143fe96874bd8980a5fb3e50f685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:03.603186 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.599309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41871 (local address 127.0.0.1:45250)
0927 02:44:03.599455 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:03.599472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:03.599578 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:03.599942 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:03.599945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:03.599952 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:03.600069 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.600075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.600796 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.600801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.601407 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.601414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.601864 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.601867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.601964 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.602004 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:03.602822 (+ 818us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:03.602846 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:44:03.602951 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 02:44:03.603056 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:44:03.603222 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.599425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.0.98.2:41871)
0927 02:44:03.599561 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.599566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.599580 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:03.599750 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.599753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.599807 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.599940 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:03.600175 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.600683 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.601535 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.601754 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.601866 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.602303 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:03.602715 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:03.602847 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.602976 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.603030 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:44:03.605966 1669 raft_consensus.cc:1261] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Refusing update from remote peer 8a7143fe96874bd8980a5fb3e50f685d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:03.606236 1949 raft_consensus.cc:1261] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c [term 1 FOLLOWER]: Refusing update from remote peer 8a7143fe96874bd8980a5fb3e50f685d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:03.606513 2059 consensus_queue.cc:1035] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:03.606724 2026 consensus_queue.cc:1035] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d [LEADER]: Connected to new peer: Peer: permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:03.608331 2036 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: TxnStatusTablet state changed. Reason: New leader 8a7143fe96874bd8980a5fb3e50f685d. Latest consensus state: current_term: 1 leader_uuid: "8a7143fe96874bd8980a5fb3e50f685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.608453 2036 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:03.608958 2024 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: TxnStatusTablet state changed. Reason: New leader 8a7143fe96874bd8980a5fb3e50f685d. Latest consensus state: current_term: 1 leader_uuid: "8a7143fe96874bd8980a5fb3e50f685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.609069 2024 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:03.610574 2026 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a7143fe96874bd8980a5fb3e50f685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.610673 2026 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:03.610756 2032 mvcc.cc:205] Tried to move back new op lower bound from 7333793970605088768 to 7333793970444603392. Current Snapshot: MvccSnapshot[applied={T|T < 7333793970605088768}]
I20260927 02:44:03.611554 2059 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a7143fe96874bd8980a5fb3e50f685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.611646 2059 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P 8a7143fe96874bd8980a5fb3e50f685d: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:03.611989 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333793970605088768 to 7333793970444603392. Current Snapshot: MvccSnapshot[applied={T|T < 7333793970605088768}]
I20260927 02:44:03.612164 2036 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a7143fe96874bd8980a5fb3e50f685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.612259 2036 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P 32f2833a7f1a4d898752bb8ae629d64c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:03.613571 2024 tablet_replica.cc:439] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a7143fe96874bd8980a5fb3e50f685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 } } peers { permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "32f2833a7f1a4d898752bb8ae629d64c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44683 } } }
I20260927 02:44:03.613685 2024 tablet_replica.cc:442] T 52fa8c3336934a5788a4d5818d5f3ded P a7dae0d0f869452cb5e0698abe94fd1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:03.625676 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.622049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52603 (local address 127.0.98.2:41871)
0927 02:44:03.622164 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.622168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.622314 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:44:03.622429 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.622431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.622468 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.622531 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:03.622773 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.623271 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.624554 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.624824 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.624949 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.625179 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:03.625282 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:03.625419 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.625538 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.625580 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:44:03.625758 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.621902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41871 (local address 127.0.98.3:52603)
0927 02:44:03.622228 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:03.622243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:03.622344 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:03.622562 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:03.622565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:03.622573 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:03.622675 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.622681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.623372 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.623376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.624431 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.624442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.624928 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.624932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.625016 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.625062 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:03.625430 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:03.625436 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:44:03.625529 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:44:03.625610 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":113,"threads_started":1}
I20260927 02:44:03.631918 1809 raft_consensus.cc:1261] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Refusing update from remote peer 32f2833a7f1a4d898752bb8ae629d64c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:03.632084 1669 raft_consensus.cc:1261] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Refusing update from remote peer 32f2833a7f1a4d898752bb8ae629d64c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:03.632268 2025 consensus_queue.cc:1035] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a7143fe96874bd8980a5fb3e50f685d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:03.632404 2036 consensus_queue.cc:1035] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7dae0d0f869452cb5e0698abe94fd1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:03.658182 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.652674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41195 (local address 127.0.98.3:44683)
0927 02:44:03.652776 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.652781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.652961 (+ 180us) server_negotiation.cc:407] Connection header received
0927 02:44:03.653061 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.653064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.653115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.653193 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:03.653399 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.654195 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.656888 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.657226 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.657326 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.657751 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:03.657831 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:03.657920 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.658005 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.658062 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:44:03.658228 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.652604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44683 (local address 127.0.98.2:41195)
0927 02:44:03.652900 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:03.652912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:03.652993 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:03.653204 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:03.653206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:03.653211 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:03.653310 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.653314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.654304 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.654308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.656767 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:03.656779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.657324 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.657328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:03.657421 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.657601 (+ 180us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:03.657921 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:03.657927 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:44:03.657997 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:44:03.658075 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":108,"threads_started":1}
I20260927 02:44:03.682523 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:03.682687 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:03.684639 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda665757d35425ba2206dc0252e5c2f: Sending DeleteTablet for 3 replicas of tablet 88541950438b42a99e8d8f83be524796
I20260927 02:44:03.685194 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 88541950438b42a99e8d8f83be524796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42936
I20260927 02:44:03.685257 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 88541950438b42a99e8d8f83be524796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50522
I20260927 02:44:03.685364 2042 tablet_replica.cc:330] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: stopping tablet replica
I20260927 02:44:03.685420 2043 tablet_replica.cc:330] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: stopping tablet replica
I20260927 02:44:03.685470 2042 raft_consensus.cc:2229] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:03.685530 2043 raft_consensus.cc:2229] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:03.685582 2042 raft_consensus.cc:2258] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:03.685695 2043 raft_consensus.cc:2258] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:03.686049 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 88541950438b42a99e8d8f83be524796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45236
I20260927 02:44:03.686300 2041 tablet_replica.cc:330] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: stopping tablet replica
I20260927 02:44:03.686430 2041 raft_consensus.cc:2229] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:03.686545 2041 raft_consensus.cc:2258] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:03.686551 2042 ts_tablet_manager.cc:1905] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:03.687477 2041 ts_tablet_manager.cc:1905] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:03.690013 2042 ts_tablet_manager.cc:1918] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:03.690088 2042 log.cc:1196] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/88541950438b42a99e8d8f83be524796
I20260927 02:44:03.690251 2041 ts_tablet_manager.cc:1918] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:03.690325 2041 log.cc:1196] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/88541950438b42a99e8d8f83be524796
I20260927 02:44:03.690337 2042 ts_tablet_manager.cc:1939] T 88541950438b42a99e8d8f83be524796 P a7dae0d0f869452cb5e0698abe94fd1d: Deleting consensus metadata
I20260927 02:44:03.690599 2041 ts_tablet_manager.cc:1939] T 88541950438b42a99e8d8f83be524796 P 8a7143fe96874bd8980a5fb3e50f685d: Deleting consensus metadata
I20260927 02:44:03.690996 1499 catalog_manager.cc:4841] TS a7dae0d0f869452cb5e0698abe94fd1d (127.0.98.1:33187): tablet 88541950438b42a99e8d8f83be524796 (table test-table [id=64a07492881b4c4fafb5b7ba22c5d954]) successfully deleted
I20260927 02:44:03.691149 1499 catalog_manager.cc:4841] TS 8a7143fe96874bd8980a5fb3e50f685d (127.0.98.2:41871): tablet 88541950438b42a99e8d8f83be524796 (table test-table [id=64a07492881b4c4fafb5b7ba22c5d954]) successfully deleted
I20260927 02:44:03.691811 2043 ts_tablet_manager.cc:1905] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 02:44:03 dist-test-slave-gkrl krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:44:03.694818 2043 ts_tablet_manager.cc:1918] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:03.694893 2043 log.cc:1196] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/88541950438b42a99e8d8f83be524796
I20260927 02:44:03.695199 2043 ts_tablet_manager.cc:1939] T 88541950438b42a99e8d8f83be524796 P 32f2833a7f1a4d898752bb8ae629d64c: Deleting consensus metadata
I20260927 02:44:03.695834 1497 catalog_manager.cc:4841] TS 32f2833a7f1a4d898752bb8ae629d64c (127.0.98.3:44683): tablet 88541950438b42a99e8d8f83be524796 (table test-table [id=64a07492881b4c4fafb5b7ba22c5d954]) successfully deleted
I20260927 02:44:03.698446 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.686788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.0.98.1:33187)
0927 02:44:03.686904 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.686908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.687183 (+ 275us) server_negotiation.cc:407] Connection header received
0927 02:44:03.687371 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.687374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.687424 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.687488 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:03.688485 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.689233 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.690404 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.690801 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.690880 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.694287 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:03.694298 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:03.694314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:03.694387 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:03.696490 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:03.696968 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:03.696978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:03.696981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:03.697076 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:03.697350 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:03.697353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:03.697355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:03.697611 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:03.697762 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.698178 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.698290 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 02:44:03.699244 1709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42948
I20260927 02:44:03.709486 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:03.700818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.0.98.62:36193)
0927 02:44:03.700933 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:03.700937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:03.701427 (+ 490us) server_negotiation.cc:407] Connection header received
0927 02:44:03.701561 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:03.701564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:03.701613 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:03.701689 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:03.702431 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.703106 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.704221 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:03.704432 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:03.704513 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:03.706164 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:03.706167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:03.706169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:03.706205 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:03.707750 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:03.708315 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:03.708318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:03.708319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:03.708395 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:03.708720 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:03.708723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:03.708724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:03.708902 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:03.709003 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:03.709253 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:44:03.709337 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 02:44:03.709955 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39264
I20260927 02:44:03.711230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 1584
I20260927 02:44:03.718379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 1724
I20260927 02:44:03.724969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 1864
I20260927 02:44:03.731123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 1480
2026-09-27T02:44:03Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4433 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:03 dist-test-slave-gkrl krb5kdc[2081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:03 dist-test-slave-gkrl krb5kdc[2081](info): set up 2 sockets
Sep 27 02:44:03 dist-test-slave-gkrl krb5kdc[2081](info): commencing operation
krb5kdc: starting...
W20260927 02:44:04.771728 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:04 dist-test-slave-gkrl krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:04.890290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:04.995532 2094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:04.995779 2094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:04.995817 2094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:05.000900 2094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:05.000962 2094 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:44:05.000985 2094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:05.001010 2094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:05.001032 2094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:05.001055 2094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:05.006491 2094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33503
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:05.007438 2094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:05.008514 2094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:05.016583 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:05.016641 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:05.016645 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:05.016850 2094 server_base.cc:1034] running on GCE node
I20260927 02:44:05.017472 2094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:05.018453 2094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:05.019610 2094 hybrid_clock.cc:648] HybridClock initialized: now 1790477045019584 us; error 39 us; skew 500 ppm
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:05.022203 2094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:05.023062 2094 webserver.cc:460] Webserver started at http://127.0.98.62:36355/ using document root <none> and password file <none>
I20260927 02:44:05.023597 2094 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:05.023692 2094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:05.023989 2094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:05.025942 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "5e97e5351a6e4735a0732372ba8d9e75"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "fbf98a3edb028286ba25e874ed0b5d83"
server_key_iv: "f3b1e848c8fd187281bd9656be2c427a"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.026490 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e97e5351a6e4735a0732372ba8d9e75"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "fbf98a3edb028286ba25e874ed0b5d83"
server_key_iv: "f3b1e848c8fd187281bd9656be2c427a"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.030081 2094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:05.032364 2109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:05.033511 2094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:05.033654 2094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "5e97e5351a6e4735a0732372ba8d9e75"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "fbf98a3edb028286ba25e874ed0b5d83"
server_key_iv: "f3b1e848c8fd187281bd9656be2c427a"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.033763 2094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:05.047811 2094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:05.050469 2094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:05.050666 2094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:05.058507 2094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33503
I20260927 02:44:05.058524 2171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33503 every 8 connection(s)
I20260927 02:44:05.059402 2094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:05.061988 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:05.065605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 2094
I20260927 02:44:05.065724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:05.065987 392 external_mini_cluster.cc:1428] Setting key d1d3a014f128a8ac900fc25ec72177a9
I20260927 02:44:05.067629 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: Bootstrap starting.
I20260927 02:44:05.069779 2172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:05.070493 2172 log.cc:824] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:05.072631 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: No bootstrap required, opened a new log
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:05.077121 2172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e97e5351a6e4735a0732372ba8d9e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33503 } }
I20260927 02:44:05.077381 2172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:05.077442 2172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e97e5351a6e4735a0732372ba8d9e75, State: Initialized, Role: FOLLOWER
I20260927 02:44:05.077894 2172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e97e5351a6e4735a0732372ba8d9e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33503 } }
I20260927 02:44:05.078015 2172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:05.078107 2172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:05.078197 2172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:05.079280 2172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e97e5351a6e4735a0732372ba8d9e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33503 } }
I20260927 02:44:05.079612 2172 leader_election.cc:302] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e97e5351a6e4735a0732372ba8d9e75; no voters:
I20260927 02:44:05.079939 2172 leader_election.cc:288] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:05.080052 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:05.080302 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [term 1 LEADER]: Becoming Leader. State: Replica: 5e97e5351a6e4735a0732372ba8d9e75, State: Running, Role: LEADER
I20260927 02:44:05.080590 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e97e5351a6e4735a0732372ba8d9e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33503 } }
I20260927 02:44:05.080749 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.067346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.0.98.62:33503)
0927 02:44:05.067806 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.067821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.067856 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:44:05.068533 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.068548 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.068868 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.069204 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.069967 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.071078 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.072298 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.072785 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.072913 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.075684 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.075689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.075705 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.075734 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.077652 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.078101 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.078107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.078114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.078250 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.078580 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.078583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.078585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.079052 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.079261 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.079438 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.079732 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
I20260927 02:44:05.081153 2172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:05.082298 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e97e5351a6e4735a0732372ba8d9e75. Latest consensus state: current_term: 1 leader_uuid: "5e97e5351a6e4735a0732372ba8d9e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e97e5351a6e4735a0732372ba8d9e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33503 } } }
I20260927 02:44:05.082316 2178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e97e5351a6e4735a0732372ba8d9e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e97e5351a6e4735a0732372ba8d9e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33503 } } }
I20260927 02:44:05.082428 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:05.082430 2178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:05.082933 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:05.086030 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:05.091831 2182 catalog_manager.cc:1347] Generated new cluster ID: 09df01448ab94f6bb4483602691d6cac
I20260927 02:44:05.091929 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:05.099332 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:05.100224 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:05.105187 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: Generated new TSK 0
I20260927 02:44:05.105762 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:05.133360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33503
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:05.243844 2198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:05.244184 2198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:05.244284 2198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:05.248256 2198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:05.248329 2198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:05.248392 2198 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:44:05.248433 2198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:05.248503 2198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:05.253675 2198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:05.255048 2198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:05.256148 2198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:05.265277 2203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:05.265400 2198 server_base.cc:1034] running on GCE node
W20260927 02:44:05.265260 2204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:05.265237 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:05.265944 2198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:05.266567 2198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:05.267747 2198 hybrid_clock.cc:648] HybridClock initialized: now 1790477045267713 us; error 47 us; skew 500 ppm
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:05.270427 2198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:05.271217 2198 webserver.cc:460] Webserver started at http://127.0.98.1:35471/ using document root <none> and password file <none>
I20260927 02:44:05.271705 2198 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:05.271761 2198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:05.271975 2198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:05.273916 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "f76b2f758ce54157887a14bdaef369b5"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "1e05327a5b56f3a1782aedad8d42d199"
server_key_iv: "47b2844e4902544316955f3531c4874a"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.274420 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f76b2f758ce54157887a14bdaef369b5"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "1e05327a5b56f3a1782aedad8d42d199"
server_key_iv: "47b2844e4902544316955f3531c4874a"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.277844 2198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:05.280145 2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:05.281461 2198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:05.281567 2198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "f76b2f758ce54157887a14bdaef369b5"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "1e05327a5b56f3a1782aedad8d42d199"
server_key_iv: "47b2844e4902544316955f3531c4874a"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.281661 2198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:05.297814 2198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:05.301031 2198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:05.301219 2198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:05.302735 2198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:05.302781 2198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:05.302817 2198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:05.302868 2198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:05.312644 2198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34495
I20260927 02:44:05.312672 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34495 every 8 connection(s)
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:05.313517 2198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:05.317021 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.303424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50571 (local address 127.0.98.62:33503)
0927 02:44:05.303579 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.303584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.304875 (+ 1291us) server_negotiation.cc:407] Connection header received
0927 02:44:05.306158 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.306161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.306218 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.306303 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.307836 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.308824 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.310241 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.310610 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.310749 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.313597 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.313603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.313609 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.313636 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.315078 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.315523 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.315526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.315527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.315598 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.316062 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.316065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.316066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.316246 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.316334 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.316789 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.316897 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:44:05.317713 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.304284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33503 (local address 127.0.98.1:50571)
0927 02:44:05.304731 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.304779 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.305924 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.306488 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.306501 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.307000 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:05.307694 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.307714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.308977 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.308982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.310118 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.310127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.310803 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.310808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.311051 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.311695 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:05.311713 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:05.313428 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:05.315180 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.315185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.315194 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.315434 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.315682 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.315684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.315685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.315806 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.316343 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:05.316348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:05.316553 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.316710 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.316936 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":120,"threads_started":1}
I20260927 02:44:05.319202 2329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33503
I20260927 02:44:05.319254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 2198
I20260927 02:44:05.319352 2329 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:05.319407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:05.319667 392 external_mini_cluster.cc:1428] Setting key 342f1850717cd98b5200c787a768fbb3
I20260927 02:44:05.319931 2329 heartbeater.cc:511] Master 127.0.98.62:33503 requested a full tablet report, sending...
I20260927 02:44:05.321501 2125 ts_manager.cc:194] Registered new tserver with Master: f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495)
I20260927 02:44:05.323057 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50571
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:44:05.330775 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.321340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35455 (local address 127.0.98.62:33503)
0927 02:44:05.323308 (+ 1968us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.323315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.323332 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:44:05.323371 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.323375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.323428 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.323514 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.324044 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.324775 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.326098 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.326312 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.326387 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.327624 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.327626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.327628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.327649 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.329095 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.329599 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.329604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.329606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.329691 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.329972 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.329979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.329980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.330160 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.330262 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.330526 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.330629 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1878}
I20260927 02:44:05.330786 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.321297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33503 (local address 127.0.98.1:35455)
0927 02:44:05.322940 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.322956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.323037 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.323547 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.323550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.323764 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:05.323922 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.323933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.324917 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.324924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.325958 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.325972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.326426 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.326440 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.326564 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.327044 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:05.327061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:05.327494 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:05.329238 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.329244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.329248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.329491 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.329789 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.329793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.329795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.329883 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.330265 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:05.330270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:05.330362 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.330465 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.330598 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:05.346796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33503
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:05.452517 2338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:05.452744 2338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:05.452788 2338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:05.456743 2338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:05.456789 2338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:05.456827 2338 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:44:05.456840 2338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:05.456871 2338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:05.461987 2338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:05.462816 2338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:05.463677 2338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:05.472724 2346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:05.472816 2344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:05.472723 2343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:05.473192 2338 server_base.cc:1034] running on GCE node
I20260927 02:44:05.473522 2338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:05.474067 2338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:05.475294 2338 hybrid_clock.cc:648] HybridClock initialized: now 1790477045475275 us; error 37 us; skew 500 ppm
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:05.478003 2338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:05.478814 2338 webserver.cc:460] Webserver started at http://127.0.98.2:40835/ using document root <none> and password file <none>
I20260927 02:44:05.479348 2338 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:05.479435 2338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:05.479655 2338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:05.481773 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "11d0f27eb33847dfa56214c3cb911f40"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "6ae20ce362dac621a2f246ccf6e030a8"
server_key_iv: "37a495e10270c92eb4bf9b48a3b09da0"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.482404 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "11d0f27eb33847dfa56214c3cb911f40"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "6ae20ce362dac621a2f246ccf6e030a8"
server_key_iv: "37a495e10270c92eb4bf9b48a3b09da0"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.486089 2338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:05.488380 2353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:05.489534 2338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:05.489660 2338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "11d0f27eb33847dfa56214c3cb911f40"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "6ae20ce362dac621a2f246ccf6e030a8"
server_key_iv: "37a495e10270c92eb4bf9b48a3b09da0"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.489773 2338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:05.500976 2338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:05.503553 2338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:05.503751 2338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:05.505066 2338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:05.505152 2338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:05.505244 2338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:05.505296 2338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:05.516376 2338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42755
I20260927 02:44:05.516532 2468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42755 every 8 connection(s)
I20260927 02:44:05.517234 2338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:05.518857 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.505999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35581 (local address 127.0.98.62:33503)
0927 02:44:05.506143 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.506147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.507111 (+ 964us) server_negotiation.cc:407] Connection header received
0927 02:44:05.508236 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.508241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.508305 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.508382 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.509909 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.510563 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.511805 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.512126 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.512268 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.515264 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.515268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.515270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.515294 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.516924 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.517401 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.517405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.517407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.517501 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.517944 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.517948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.517949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.518171 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.518249 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.518649 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.518743 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 02:44:05.519601 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.506478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33503 (local address 127.0.98.2:35581)
0927 02:44:05.506964 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.507002 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.507985 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.508803 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.508810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.509228 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:05.509782 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.509801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.510680 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.510683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.511686 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.511695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.512314 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.512323 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.512559 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.513191 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:05.513220 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:05.515101 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:05.517039 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.517044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.517057 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.517295 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.517638 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.517642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.517644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.517817 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.518245 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:05.518249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:05.518444 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.518610 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.518829 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":137,"threads_started":1}
I20260927 02:44:05.520861 2469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33503
I20260927 02:44:05.520975 2469 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:05.521365 2469 heartbeater.cc:511] Master 127.0.98.62:33503 requested a full tablet report, sending...
I20260927 02:44:05.522265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 2338
I20260927 02:44:05.522379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:05.522373 2126 ts_manager.cc:194] Registered new tserver with Master: 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755)
I20260927 02:44:05.522655 392 external_mini_cluster.cc:1428] Setting key 40c826c948f0ec0b88d86ce6dcca1a82
I20260927 02:44:05.523118 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35581
I20260927 02:44:05.530313 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.523129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51849 (local address 127.0.98.62:33503)
0927 02:44:05.523240 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.523244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.523258 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:05.523484 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.523487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.523528 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.523618 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.524110 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.524815 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.525897 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.526122 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.526228 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.527264 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.527268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.527270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.527293 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.528735 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.529139 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.529144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.529146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.529224 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.529503 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.529506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.529508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.529710 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.529816 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.530061 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.530164 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:44:05.530332 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.522827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33503 (local address 127.0.98.2:51849)
0927 02:44:05.523092 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.523108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.523364 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.523631 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.523634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.523820 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:05.523983 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.523992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.524954 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.524960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.525777 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.525786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.526241 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.526245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.526339 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.526817 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:05.526833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:05.527124 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:05.528845 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.528850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.528852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.529042 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.529314 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.529319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.529321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.529406 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.529806 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:05.529810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:05.529903 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.530010 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.530155 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:05.548412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33503
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:05.655120 2478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:05.655345 2478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:05.655390 2478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:05.659320 2478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:05.659372 2478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:05.659446 2478 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:44:05.659478 2478 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:05.659569 2478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:05.664685 2478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:05.665601 2478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:05.666527 2478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:05.675781 2483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:05.675815 2486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:05.675806 2478 server_base.cc:1034] running on GCE node
W20260927 02:44:05.675920 2484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:05.676452 2478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:05.676992 2478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:05.678184 2478 hybrid_clock.cc:648] HybridClock initialized: now 1790477045678150 us; error 50 us; skew 500 ppm
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:05.680892 2478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:05.681790 2478 webserver.cc:460] Webserver started at http://127.0.98.3:35369/ using document root <none> and password file <none>
I20260927 02:44:05.682371 2478 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:05.682457 2478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:05.682715 2478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:05.684890 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "63706950adfc47b68ba99f311ca25fb8"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "1437a40ea3072131eff32dccb4c2d0f7"
server_key_iv: "178202a5da0235266ada4ed9ae8c461c"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.685441 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "63706950adfc47b68ba99f311ca25fb8"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "1437a40ea3072131eff32dccb4c2d0f7"
server_key_iv: "178202a5da0235266ada4ed9ae8c461c"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.689076 2478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:05.691341 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:05.692523 2478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:05.692655 2478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "63706950adfc47b68ba99f311ca25fb8"
format_stamp: "Formatted at 2026-09-27 02:44:05 on dist-test-slave-gkrl"
server_key: "1437a40ea3072131eff32dccb4c2d0f7"
server_key_iv: "178202a5da0235266ada4ed9ae8c461c"
server_key_version: "encryptionkey@0"
I20260927 02:44:05.692759 2478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:05.707329 2478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:05.710362 2478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:05.710557 2478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:05.711941 2478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:05.712016 2478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:05.712082 2478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:05.712128 2478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:05.722872 2478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40283
I20260927 02:44:05.723281 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40283 every 8 connection(s)
I20260927 02:44:05.723866 2478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:05.726171 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.712847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35783 (local address 127.0.98.62:33503)
0927 02:44:05.713004 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.713008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.713883 (+ 875us) server_negotiation.cc:407] Connection header received
0927 02:44:05.715151 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.715158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.715229 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.715296 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.717136 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.717793 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.719090 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.719493 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.719618 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.722284 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.722288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.722291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.722317 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.724013 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.724686 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.724690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.724691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.724787 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.725135 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.725138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.725140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.725326 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.725434 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.725891 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.726005 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:44:05.726892 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.713248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33503 (local address 127.0.98.3:35783)
0927 02:44:05.713712 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.713770 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.714881 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.715749 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.715759 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.716264 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:05.716985 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.717006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.717930 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.717933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.718901 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.718909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.719674 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.719689 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.719914 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.720501 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:05.720517 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:05.722123 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:05.724145 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.724151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.724164 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.724508 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.724885 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.724890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.724892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.725028 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.725432 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:05.725438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:05.725669 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.725848 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.726135 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260927 02:44:05.728127 2609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33503
I20260927 02:44:05.728272 2609 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:05.728742 2609 heartbeater.cc:511] Master 127.0.98.62:33503 requested a full tablet report, sending...
I20260927 02:44:05.729677 2125 ts_manager.cc:194] Registered new tserver with Master: 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283)
I20260927 02:44:05.730304 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35783
I20260927 02:44:05.733978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 2478
I20260927 02:44:05.734099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:05.734356 392 external_mini_cluster.cc:1428] Setting key 3e1d8e24892d0b1bc5d907e69ee8fadd
I20260927 02:44:05.736202 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:05.737473 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.730247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33095 (local address 127.0.98.62:33503)
0927 02:44:05.730371 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.730375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.730585 (+ 210us) server_negotiation.cc:407] Connection header received
0927 02:44:05.730692 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.730696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.730733 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.730823 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.731227 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.731716 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.732524 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.732752 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.732859 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.733880 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.733883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.733886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.733907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.735676 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.736190 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.736196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.736198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.736283 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.736594 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.736598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.736600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.736786 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.736893 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.737196 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.737302 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:44:05.737485 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.730175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33503 (local address 127.0.98.3:33095)
0927 02:44:05.730506 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.730517 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.730621 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.730842 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.730845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.731035 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:05.731121 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.731126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.731832 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.731836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.732410 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.732418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.732862 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.732865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.732959 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.733439 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:05.733453 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:05.733749 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:05.735821 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.735824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.735826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.736083 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.736389 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.736394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.736396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.736489 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.736895 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:05.736899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:05.737022 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.737139 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.737283 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:05.760993 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.753212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.98.62:33503)
0927 02:44:05.753342 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.753345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.753358 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:05.753395 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.753397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.753446 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.753515 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.754177 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.754687 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.755475 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.755685 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.755776 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.758006 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.758009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.758011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.758032 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.759549 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.759998 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.760000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.760002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.760075 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.760319 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.760321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.760322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.760502 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.760597 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.760795 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.760880 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:44:05.763801 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:05.766799 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:05.780972 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.775770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34495 (local address 127.0.0.1:43364)
0927 02:44:05.776579 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.776595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.776704 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.777182 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.777185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.777201 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:05.777280 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.777286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.778575 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.778578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.779496 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.779506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.780425 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.780429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.780527 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.780562 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.780703 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.780823 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"spinlock_wait_cycles":9728,"thread_start_us":112,"threads_started":1}
I20260927 02:44:05.781031 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.776195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.0.98.1:34495)
0927 02:44:05.776533 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.776545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.776794 (+ 249us) server_negotiation.cc:407] Connection header received
0927 02:44:05.776871 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.776876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.777044 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.777189 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:05.777393 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.778458 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.779671 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.780303 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.780427 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.780583 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.780740 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.780828 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":96,"threads_started":1}
I20260927 02:44:05.781136 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.774923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42755 (local address 127.0.0.1:58114)
0927 02:44:05.775252 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.775301 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.775411 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.776569 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.776575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.776598 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:05.776753 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.776760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.777949 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.777954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.779538 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.779548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.780679 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.780683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.780772 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.780802 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.780852 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.780919 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":85,"thread_start_us":74,"threads_started":1}
I20260927 02:44:05.781416 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.775121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40283 (local address 127.0.0.1:39730)
0927 02:44:05.775629 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.775642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.775716 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.776667 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.776669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.776693 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:05.776810 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.776817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.778161 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.778164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.779548 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.779554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.780828 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.780835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.780924 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.780953 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.781019 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.781283 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":111,"spinlock_wait_cycles":896,"thread_start_us":106,"threads_started":1}
I20260927 02:44:05.781795 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.775098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.0.98.2:42755)
0927 02:44:05.776040 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.776054 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.776079 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:44:05.776157 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.776164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.776395 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.776900 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:05.777026 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.777842 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.779675 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.780509 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.781281 (+ 772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.781453 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.781514 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.781616 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":99,"threads_started":1}
I20260927 02:44:05.782064 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.775794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.0.98.3:40283)
0927 02:44:05.776223 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.776236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.776259 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:44:05.776336 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.776342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.776528 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.776707 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:05.776899 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.778048 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.779676 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.780498 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.780614 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.780774 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.781734 (+ 960us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.781834 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":120,"threads_started":1}
I20260927 02:44:05.784586 2543 tablet_service.cc:1467] Processing CreateTablet for tablet d188a00f9adc47c39df909f512c9562c (DEFAULT_TABLE table=test-table [id=33b736840dac4e8abe02d1cf101444e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:05.784647 2263 tablet_service.cc:1467] Processing CreateTablet for tablet d188a00f9adc47c39df909f512c9562c (DEFAULT_TABLE table=test-table [id=33b736840dac4e8abe02d1cf101444e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:05.784776 2403 tablet_service.cc:1467] Processing CreateTablet for tablet d188a00f9adc47c39df909f512c9562c (DEFAULT_TABLE table=test-table [id=33b736840dac4e8abe02d1cf101444e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:05.785632 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188a00f9adc47c39df909f512c9562c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:05.785632 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188a00f9adc47c39df909f512c9562c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:05.785647 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188a00f9adc47c39df909f512c9562c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:05.791911 2632 tablet_bootstrap.cc:492] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: Bootstrap starting.
I20260927 02:44:05.793197 2634 tablet_bootstrap.cc:492] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: Bootstrap starting.
I20260927 02:44:05.793226 2633 tablet_bootstrap.cc:492] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Bootstrap starting.
I20260927 02:44:05.794659 2632 tablet_bootstrap.cc:654] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:05.794873 2633 tablet_bootstrap.cc:654] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:05.795130 2634 tablet_bootstrap.cc:654] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:05.795540 2632 log.cc:824] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:05.795686 2633 log.cc:824] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:05.795928 2634 log.cc:824] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:05.797425 2633 tablet_bootstrap.cc:492] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: No bootstrap required, opened a new log
I20260927 02:44:05.797488 2632 tablet_bootstrap.cc:492] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: No bootstrap required, opened a new log
I20260927 02:44:05.797623 2633 ts_tablet_manager.cc:1397] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:44:05.797642 2634 tablet_bootstrap.cc:492] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: No bootstrap required, opened a new log
I20260927 02:44:05.797676 2632 ts_tablet_manager.cc:1397] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260927 02:44:05.797780 2634 ts_tablet_manager.cc:1397] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:44:05.800426 2633 raft_consensus.cc:348] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.800647 2633 raft_consensus.cc:374] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:05.800705 2633 raft_consensus.cc:729] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76b2f758ce54157887a14bdaef369b5, State: Initialized, Role: FOLLOWER
I20260927 02:44:05.801129 2633 consensus_queue.cc:260] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.801461 2632 raft_consensus.cc:348] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.801713 2632 raft_consensus.cc:374] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:05.801740 2329 heartbeater.cc:503] Master 127.0.98.62:33503 was elected leader, sending a full tablet report...
I20260927 02:44:05.801782 2632 raft_consensus.cc:729] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63706950adfc47b68ba99f311ca25fb8, State: Initialized, Role: FOLLOWER
I20260927 02:44:05.802124 2633 ts_tablet_manager.cc:1428] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:44:05.802168 2632 consensus_queue.cc:260] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.801461 2634 raft_consensus.cc:348] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.802873 2634 raft_consensus.cc:374] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:05.802933 2634 raft_consensus.cc:729] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d0f27eb33847dfa56214c3cb911f40, State: Initialized, Role: FOLLOWER
I20260927 02:44:05.803308 2632 ts_tablet_manager.cc:1428] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 02:44:05.803311 2634 consensus_queue.cc:260] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.803905 2609 heartbeater.cc:503] Master 127.0.98.62:33503 was elected leader, sending a full tablet report...
I20260927 02:44:05.804181 2634 ts_tablet_manager.cc:1428] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:44:05.804472 2469 heartbeater.cc:503] Master 127.0.98.62:33503 was elected leader, sending a full tablet report...
W20260927 02:44:05.814615 2330 tablet.cc:2369] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:05.869596 2639 raft_consensus.cc:482] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:05.869809 2639 raft_consensus.cc:504] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.870906 2639 leader_election.cc:288] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755), f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495)
I20260927 02:44:05.875409 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.871109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42755 (local address 127.0.98.3:58313)
0927 02:44:05.871240 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.871259 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.871344 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.871700 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.871704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.871720 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:05.871846 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.871855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.872867 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.872870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.874217 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.874228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.874940 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.874943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.875072 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.875093 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.875154 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.875259 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 02:44:05.875411 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.871169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58313 (local address 127.0.98.2:42755)
0927 02:44:05.871334 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.871339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.871355 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:05.871464 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.871468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.871532 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.871631 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:05.871970 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.872753 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.874343 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.874815 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.874919 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.874980 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.875209 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.875278 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 02:44:05.875865 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.871376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33383 (local address 127.0.98.1:34495)
0927 02:44:05.871487 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.871492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.871628 (+ 136us) server_negotiation.cc:407] Connection header received
0927 02:44:05.871775 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.871779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.871834 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.871880 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:05.872256 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.872952 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.874665 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.875255 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.875357 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.875412 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.875663 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.875755 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:44:05.876065 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d188a00f9adc47c39df909f512c9562c" candidate_uuid: "63706950adfc47b68ba99f311ca25fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d0f27eb33847dfa56214c3cb911f40" is_pre_election: true
I20260927 02:44:05.876222 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.871135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34495 (local address 127.0.98.3:33383)
0927 02:44:05.871535 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.871546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.871658 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.872040 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.872041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.872056 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:05.872147 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.872152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.873047 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.873050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.874549 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.874556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.875368 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.875371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.875498 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.875522 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.876013 (+ 491us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.876097 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":131,"threads_started":1}
I20260927 02:44:05.876372 2423 raft_consensus.cc:2454] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63706950adfc47b68ba99f311ca25fb8 in term 0.
I20260927 02:44:05.876720 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d188a00f9adc47c39df909f512c9562c" candidate_uuid: "63706950adfc47b68ba99f311ca25fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76b2f758ce54157887a14bdaef369b5" is_pre_election: true
I20260927 02:44:05.876873 2496 leader_election.cc:302] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, 63706950adfc47b68ba99f311ca25fb8; no voters:
I20260927 02:44:05.876986 2283 raft_consensus.cc:2454] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63706950adfc47b68ba99f311ca25fb8 in term 0.
I20260927 02:44:05.877198 2639 raft_consensus.cc:2781] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:05.877287 2639 raft_consensus.cc:482] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:05.877348 2639 raft_consensus.cc:3037] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:05.878367 2639 raft_consensus.cc:504] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.878671 2639 leader_election.cc:288] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755), f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495)
I20260927 02:44:05.878962 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d188a00f9adc47c39df909f512c9562c" candidate_uuid: "63706950adfc47b68ba99f311ca25fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76b2f758ce54157887a14bdaef369b5"
I20260927 02:44:05.878990 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d188a00f9adc47c39df909f512c9562c" candidate_uuid: "63706950adfc47b68ba99f311ca25fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d0f27eb33847dfa56214c3cb911f40"
I20260927 02:44:05.879083 2423 raft_consensus.cc:3037] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:05.879083 2283 raft_consensus.cc:3037] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:05.880291 2283 raft_consensus.cc:2454] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63706950adfc47b68ba99f311ca25fb8 in term 1.
I20260927 02:44:05.880291 2423 raft_consensus.cc:2454] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63706950adfc47b68ba99f311ca25fb8 in term 1.
I20260927 02:44:05.880571 2496 leader_election.cc:302] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, 63706950adfc47b68ba99f311ca25fb8, f76b2f758ce54157887a14bdaef369b5; no voters:
I20260927 02:44:05.880760 2639 raft_consensus.cc:2781] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:05.880993 2639 raft_consensus.cc:686] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 63706950adfc47b68ba99f311ca25fb8, State: Running, Role: LEADER
I20260927 02:44:05.881299 2639 consensus_queue.cc:237] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } }
I20260927 02:44:05.884281 2125 catalog_manager.cc:5495] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "63706950adfc47b68ba99f311ca25fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:44:05 dist-test-slave-gkrl krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:05.930228 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.921636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33503 (local address 127.0.0.1:42780)
0927 02:44:05.921948 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:05.921962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:05.922096 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:05.922360 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:05.922363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:05.922805 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:05.922892 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.922899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.923645 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.923649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.924237 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:05.924246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.924669 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.924673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:05.924750 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.925364 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:05.925377 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:05.926813 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:05.928641 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.928650 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.928656 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.928870 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.929150 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:05.929153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:05.929155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:05.929235 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:05.929618 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:05.929625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:05.929751 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:44:05.929882 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 02:44:05.930031 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":137,"threads_started":1}
I20260927 02:44:05.930228 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:05.921760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.0.98.62:33503)
0927 02:44:05.921870 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:05.921875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:05.922055 (+ 180us) server_negotiation.cc:407] Connection header received
0927 02:44:05.922204 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:05.922208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:05.922253 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:05.922338 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:05.923020 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.923530 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.924356 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:05.924564 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:05.924671 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:05.926947 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:05.926950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:05.926997 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:05.927016 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:05.928536 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.928976 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.928979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.928981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.929056 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:05.929346 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:05.929350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:05.929352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:05.929516 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:05.929620 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:05.929926 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 02:44:05.930082 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":32}
I20260927 02:44:05.934154 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42780:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:44:05.934897 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:05.939841 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 946ec8d943ad4dd4927b7b0771904dd0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e43b62f0f4442e6903d767af929bf9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:05.939906 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 946ec8d943ad4dd4927b7b0771904dd0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e43b62f0f4442e6903d767af929bf9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:05.940590 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946ec8d943ad4dd4927b7b0771904dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:05.940589 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946ec8d943ad4dd4927b7b0771904dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:05.940231 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 946ec8d943ad4dd4927b7b0771904dd0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e43b62f0f4442e6903d767af929bf9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:05.940876 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946ec8d943ad4dd4927b7b0771904dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:05.943464 2632 tablet_bootstrap.cc:492] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: Bootstrap starting.
I20260927 02:44:05.944018 2634 tablet_bootstrap.cc:492] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: Bootstrap starting.
I20260927 02:44:05.944494 2632 tablet_bootstrap.cc:654] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:05.944844 2634 tablet_bootstrap.cc:654] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:05.945523 2633 tablet_bootstrap.cc:492] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: Bootstrap starting.
I20260927 02:44:05.945843 2632 tablet_bootstrap.cc:492] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: No bootstrap required, opened a new log
I20260927 02:44:05.945946 2632 ts_tablet_manager.cc:1397] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 02:44:05.946305 2633 tablet_bootstrap.cc:654] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:05.946414 2632 raft_consensus.cc:348] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:05.946552 2632 raft_consensus.cc:374] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:05.946602 2632 raft_consensus.cc:729] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63706950adfc47b68ba99f311ca25fb8, State: Initialized, Role: FOLLOWER
I20260927 02:44:05.946733 2632 consensus_queue.cc:260] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:05.947005 2639 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:05.947151 2639 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:05.947073 2632 ts_tablet_manager.cc:1428] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:05.947768 2633 tablet_bootstrap.cc:492] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: No bootstrap required, opened a new log
I20260927 02:44:05.947850 2634 tablet_bootstrap.cc:492] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: No bootstrap required, opened a new log
I20260927 02:44:05.947851 2633 ts_tablet_manager.cc:1397] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:05.947956 2634 ts_tablet_manager.cc:1397] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 02:44:05.948342 2633 raft_consensus.cc:348] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:05.948364 2634 raft_consensus.cc:348] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:05.948454 2634 raft_consensus.cc:374] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:05.948467 2633 raft_consensus.cc:374] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:05.948490 2634 raft_consensus.cc:729] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d0f27eb33847dfa56214c3cb911f40, State: Initialized, Role: FOLLOWER
I20260927 02:44:05.948506 2633 raft_consensus.cc:729] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76b2f758ce54157887a14bdaef369b5, State: Initialized, Role: FOLLOWER
I20260927 02:44:05.948614 2633 consensus_queue.cc:260] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:05.948614 2634 consensus_queue.cc:260] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:05.948858 2634 ts_tablet_manager.cc:1428] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:05.948858 2633 ts_tablet_manager.cc:1428] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:05.949072 2640 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:05.949079 2638 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:05.949200 2638 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:05.949215 2640 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 02:44:05.974916 2610 tablet.cc:2369] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:44:06.018383 2470 tablet.cc:2369] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:06.019251 2638 raft_consensus.cc:482] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:06.019366 2638 raft_consensus.cc:504] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.020493 2638 leader_election.cc:288] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755), 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283)
I20260927 02:44:06.024580 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.020710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42755 (local address 127.0.98.1:56933)
0927 02:44:06.020998 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.021014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.021121 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.021363 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.021366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.021382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:06.021519 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.021525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.022446 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.022448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.023449 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.023459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.024250 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.024254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.024338 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.024358 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.024419 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.024470 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260927 02:44:06.024654 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.020816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56933 (local address 127.0.98.2:42755)
0927 02:44:06.021048 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.021053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.021074 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:44:06.021216 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.021219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.021256 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.021353 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:06.021623 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.022333 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.023578 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.024103 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.024244 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.024311 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.024456 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.024525 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 02:44:06.025274 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.020710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40283 (local address 127.0.98.1:44463)
0927 02:44:06.021261 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.021270 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.021363 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.021629 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.021632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.021656 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:06.022182 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.022191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.022995 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.022999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.023957 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.023965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.024820 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.024824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.024924 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.024942 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.025040 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.025155 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":12,"thread_start_us":78,"threads_started":1}
I20260927 02:44:06.025288 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.020961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44463 (local address 127.0.98.3:40283)
0927 02:44:06.021104 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.021108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.021335 (+ 227us) server_negotiation.cc:407] Connection header received
0927 02:44:06.021459 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.021461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.021514 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.021605 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:06.022324 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.022873 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.024068 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.024687 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.024801 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.024874 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.025076 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.025169 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:44:06.025251 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946ec8d943ad4dd4927b7b0771904dd0" candidate_uuid: "f76b2f758ce54157887a14bdaef369b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d0f27eb33847dfa56214c3cb911f40" is_pre_election: true
I20260927 02:44:06.025395 2423 raft_consensus.cc:2454] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76b2f758ce54157887a14bdaef369b5 in term 0.
I20260927 02:44:06.025763 2216 leader_election.cc:302] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, f76b2f758ce54157887a14bdaef369b5; no voters:
I20260927 02:44:06.025928 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946ec8d943ad4dd4927b7b0771904dd0" candidate_uuid: "f76b2f758ce54157887a14bdaef369b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63706950adfc47b68ba99f311ca25fb8" is_pre_election: true
I20260927 02:44:06.026072 2638 raft_consensus.cc:2781] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:06.026144 2638 raft_consensus.cc:482] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:06.026199 2638 raft_consensus.cc:3037] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:06.026167 2563 raft_consensus.cc:2454] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76b2f758ce54157887a14bdaef369b5 in term 0.
I20260927 02:44:06.027354 2638 raft_consensus.cc:504] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.027809 2638 leader_election.cc:288] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [CANDIDATE]: Term 1 election: Requested vote from peers 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755), 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283)
I20260927 02:44:06.028144 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946ec8d943ad4dd4927b7b0771904dd0" candidate_uuid: "f76b2f758ce54157887a14bdaef369b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d0f27eb33847dfa56214c3cb911f40"
I20260927 02:44:06.028196 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946ec8d943ad4dd4927b7b0771904dd0" candidate_uuid: "f76b2f758ce54157887a14bdaef369b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63706950adfc47b68ba99f311ca25fb8"
I20260927 02:44:06.028285 2423 raft_consensus.cc:3037] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:06.028307 2563 raft_consensus.cc:3037] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:06.029310 2563 raft_consensus.cc:2454] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76b2f758ce54157887a14bdaef369b5 in term 1.
I20260927 02:44:06.029313 2423 raft_consensus.cc:2454] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76b2f758ce54157887a14bdaef369b5 in term 1.
I20260927 02:44:06.029616 2214 leader_election.cc:302] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63706950adfc47b68ba99f311ca25fb8, f76b2f758ce54157887a14bdaef369b5; no voters:
I20260927 02:44:06.029842 2638 raft_consensus.cc:2781] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:06.030082 2638 raft_consensus.cc:686] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [term 1 LEADER]: Becoming Leader. State: Replica: f76b2f758ce54157887a14bdaef369b5, State: Running, Role: LEADER
I20260927 02:44:06.030413 2638 consensus_queue.cc:237] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.032598 2655 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: TxnStatusTablet state changed. Reason: New leader f76b2f758ce54157887a14bdaef369b5. Latest consensus state: current_term: 1 leader_uuid: "f76b2f758ce54157887a14bdaef369b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:06.032727 2655 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:06.033254 2657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:44:06.033449 2657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:44:06.033614 2124 catalog_manager.cc:5495] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f76b2f758ce54157887a14bdaef369b5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f76b2f758ce54157887a14bdaef369b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:06.045766 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.041821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34495 (local address 127.0.0.1:43370)
0927 02:44:06.041961 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.041978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.042070 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.042369 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.042371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.042376 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:06.042458 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.042462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.043514 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.043516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.044167 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.044177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.044732 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.044734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.044795 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.044833 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.045529 (+ 696us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.045541 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.045599 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.045663 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 02:44:06.045818 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.041951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.98.1:34495)
0927 02:44:06.042063 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.042068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.042084 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:06.042182 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.042185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.042252 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.042367 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:06.042561 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.043404 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.044294 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.044622 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.044740 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.044970 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.045422 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.045551 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.045630 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.045685 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:44:06.050498 2563 raft_consensus.cc:1261] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Refusing update from remote peer f76b2f758ce54157887a14bdaef369b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:06.050477 2423 raft_consensus.cc:1261] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Refusing update from remote peer f76b2f758ce54157887a14bdaef369b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:06.051170 2655 consensus_queue.cc:1035] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:06.051359 2638 consensus_queue.cc:1035] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:06.054466 2639 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: TxnStatusTablet state changed. Reason: New leader f76b2f758ce54157887a14bdaef369b5. Latest consensus state: current_term: 1 leader_uuid: "f76b2f758ce54157887a14bdaef369b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:06.054582 2639 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:06.054804 2640 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: TxnStatusTablet state changed. Reason: New leader f76b2f758ce54157887a14bdaef369b5. Latest consensus state: current_term: 1 leader_uuid: "f76b2f758ce54157887a14bdaef369b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:06.054935 2640 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:06.058286 2655 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f76b2f758ce54157887a14bdaef369b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:06.058408 2655 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:06.058857 2655 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f76b2f758ce54157887a14bdaef369b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:06.058945 2655 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P f76b2f758ce54157887a14bdaef369b5: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:06.059861 2639 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f76b2f758ce54157887a14bdaef369b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:06.059974 2639 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P 63706950adfc47b68ba99f311ca25fb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:06.060575 2661 mvcc.cc:205] Tried to move back new op lower bound from 7333793980616515584 to 7333793980543119360. Current Snapshot: MvccSnapshot[applied={T|T < 7333793980616515584}]
I20260927 02:44:06.064857 2660 mvcc.cc:205] Tried to move back new op lower bound from 7333793980616515584 to 7333793980543119360. Current Snapshot: MvccSnapshot[applied={T|T < 7333793980616515584 or (T in {7333793980616515584})}]
I20260927 02:44:06.064981 2640 tablet_replica.cc:439] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f76b2f758ce54157887a14bdaef369b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } } }
I20260927 02:44:06.065121 2640 tablet_replica.cc:442] T 946ec8d943ad4dd4927b7b0771904dd0 P 11d0f27eb33847dfa56214c3cb911f40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:06.067667 2662 mvcc.cc:205] Tried to move back new op lower bound from 7333793980616515584 to 7333793980543119360. Current Snapshot: MvccSnapshot[applied={T|T < 7333793980616515584}]
I20260927 02:44:06.074703 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.070375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.0.98.3:40283)
0927 02:44:06.070475 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.070479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.070493 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:06.070601 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.070604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.070659 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.070762 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:06.071050 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.071788 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.073067 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.073419 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.073521 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.073837 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.074296 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.074399 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.074516 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.074567 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:44:06.086292 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.082442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34495 (local address 127.0.98.3:34773)
0927 02:44:06.082751 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.082767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.082853 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.083191 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.083193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.083200 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:06.083298 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.083302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.084326 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.084329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.085017 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.085026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.085571 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.085573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.085662 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.085699 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.086035 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.086040 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.086090 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.086180 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260927 02:44:06.086318 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.082553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34773 (local address 127.0.98.1:34495)
0927 02:44:06.082669 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.082674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.082830 (+ 156us) server_negotiation.cc:407] Connection header received
0927 02:44:06.082998 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.083001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.083059 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.083160 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:06.083411 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.084205 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.085109 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.085467 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.085579 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.085853 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.085943 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.086023 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.086124 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.086176 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:44:06.092491 2423 raft_consensus.cc:1261] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Refusing update from remote peer 63706950adfc47b68ba99f311ca25fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:06.092767 2283 raft_consensus.cc:1261] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Refusing update from remote peer 63706950adfc47b68ba99f311ca25fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:06.093295 2642 consensus_queue.cc:1035] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:06.093489 2639 consensus_queue.cc:1035] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:06.115067 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.110998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51405 (local address 127.0.98.3:40283)
0927 02:44:06.111102 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.111107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.111672 (+ 565us) server_negotiation.cc:407] Connection header received
0927 02:44:06.111817 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.111819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.111907 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.111979 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:06.112231 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.112729 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.113848 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.114269 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.114376 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.114634 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.114713 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.114794 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.114865 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.114927 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:44:06.115266 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.110672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40283 (local address 127.0.98.1:51405)
0927 02:44:06.111575 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.111591 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.111717 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.111987 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.111989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.111997 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:06.112129 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.112134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.112835 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.112840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.113719 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.113730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.114406 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.114409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.114478 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.114516 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.114800 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.114804 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.114852 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.114909 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":101,"threads_started":1}
I20260927 02:44:06.141986 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:06.142180 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:06.144767 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: Sending DeleteTablet for 3 replicas of tablet d188a00f9adc47c39df909f512c9562c
I20260927 02:44:06.145488 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet d188a00f9adc47c39df909f512c9562c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58114
I20260927 02:44:06.145931 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet d188a00f9adc47c39df909f512c9562c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39730
I20260927 02:44:06.146555 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet d188a00f9adc47c39df909f512c9562c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43364
I20260927 02:44:06.146597 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:06.146998 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:06.147231 2679 tablet_replica.cc:330] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: stopping tablet replica
I20260927 02:44:06.147637 2679 raft_consensus.cc:2229] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:06.147950 2679 raft_consensus.cc:2258] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:06.148314 2680 tablet_replica.cc:330] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: stopping tablet replica
I20260927 02:44:06.148587 2680 raft_consensus.cc:2229] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:06.148912 2680 raft_consensus.cc:2258] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:06.149633 2681 tablet_replica.cc:330] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: stopping tablet replica
I20260927 02:44:06.149930 2681 raft_consensus.cc:2229] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:06.150112 2681 raft_consensus.cc:2258] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:06.152343 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 8c163d779a64428aa1f41c4ebd5ae129 (DEFAULT_TABLE table=test-table [id=01df85342a5c458cb9eb581a81f876e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:06.152622 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c163d779a64428aa1f41c4ebd5ae129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:06.152968 2679 ts_tablet_manager.cc:1905] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:06.155079 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 8c163d779a64428aa1f41c4ebd5ae129 (DEFAULT_TABLE table=test-table [id=01df85342a5c458cb9eb581a81f876e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:06.155313 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c163d779a64428aa1f41c4ebd5ae129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:06.156561 2633 tablet_bootstrap.cc:492] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: Bootstrap starting.
I20260927 02:44:06.157215 2679 ts_tablet_manager.cc:1918] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:06.157294 2679 log.cc:1196] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/d188a00f9adc47c39df909f512c9562c
I20260927 02:44:06.157431 2633 tablet_bootstrap.cc:654] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:06.157665 2679 ts_tablet_manager.cc:1939] T d188a00f9adc47c39df909f512c9562c P 11d0f27eb33847dfa56214c3cb911f40: Deleting consensus metadata
I20260927 02:44:06.158516 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 8c163d779a64428aa1f41c4ebd5ae129 (DEFAULT_TABLE table=test-table [id=01df85342a5c458cb9eb581a81f876e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:06.158775 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c163d779a64428aa1f41c4ebd5ae129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:06.160192 2112 catalog_manager.cc:4841] TS 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755): tablet d188a00f9adc47c39df909f512c9562c (table test-table [id=33b736840dac4e8abe02d1cf101444e4]) successfully deleted
I20260927 02:44:06.160429 2680 ts_tablet_manager.cc:1905] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:06.161623 2634 tablet_bootstrap.cc:492] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: Bootstrap starting.
I20260927 02:44:06.162519 2634 tablet_bootstrap.cc:654] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:06.162931 2632 tablet_bootstrap.cc:492] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: Bootstrap starting.
I20260927 02:44:06.162916 2680 ts_tablet_manager.cc:1918] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:06.163079 2680 log.cc:1196] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/d188a00f9adc47c39df909f512c9562c
I20260927 02:44:06.163389 2680 ts_tablet_manager.cc:1939] T d188a00f9adc47c39df909f512c9562c P 63706950adfc47b68ba99f311ca25fb8: Deleting consensus metadata
I20260927 02:44:06.163780 2632 tablet_bootstrap.cc:654] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:06.164623 2634 tablet_bootstrap.cc:492] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: No bootstrap required, opened a new log
I20260927 02:44:06.164641 2110 catalog_manager.cc:4841] TS 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283): tablet d188a00f9adc47c39df909f512c9562c (table test-table [id=33b736840dac4e8abe02d1cf101444e4]) successfully deleted
I20260927 02:44:06.164696 2634 ts_tablet_manager.cc:1397] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 02:44:06.165081 2634 raft_consensus.cc:348] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.165184 2634 raft_consensus.cc:374] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:06.165215 2634 raft_consensus.cc:729] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d0f27eb33847dfa56214c3cb911f40, State: Initialized, Role: FOLLOWER
I20260927 02:44:06.165326 2634 consensus_queue.cc:260] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.165434 2633 tablet_bootstrap.cc:492] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: No bootstrap required, opened a new log
I20260927 02:44:06.165505 2633 ts_tablet_manager.cc:1397] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260927 02:44:06.165565 2634 ts_tablet_manager.cc:1428] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:06.165894 2633 raft_consensus.cc:348] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.166002 2633 raft_consensus.cc:374] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:06.166033 2633 raft_consensus.cc:729] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76b2f758ce54157887a14bdaef369b5, State: Initialized, Role: FOLLOWER
I20260927 02:44:06.166139 2633 consensus_queue.cc:260] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.166384 2633 ts_tablet_manager.cc:1428] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:06.167451 2681 ts_tablet_manager.cc:1905] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:06.169605 2632 tablet_bootstrap.cc:492] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: No bootstrap required, opened a new log
I20260927 02:44:06.169693 2632 ts_tablet_manager.cc:1397] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 02:44:06.170094 2632 raft_consensus.cc:348] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.170204 2632 raft_consensus.cc:374] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:06.170239 2632 raft_consensus.cc:729] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63706950adfc47b68ba99f311ca25fb8, State: Initialized, Role: FOLLOWER
I20260927 02:44:06.170367 2632 consensus_queue.cc:260] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.170639 2632 ts_tablet_manager.cc:1428] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:06.171276 2681 ts_tablet_manager.cc:1918] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:06.171409 2681 log.cc:1196] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/d188a00f9adc47c39df909f512c9562c
I20260927 02:44:06.171765 2681 ts_tablet_manager.cc:1939] T d188a00f9adc47c39df909f512c9562c P f76b2f758ce54157887a14bdaef369b5: Deleting consensus metadata
I20260927 02:44:06.172928 2113 catalog_manager.cc:4841] TS f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495): tablet d188a00f9adc47c39df909f512c9562c (table test-table [id=33b736840dac4e8abe02d1cf101444e4]) successfully deleted
I20260927 02:44:06.405040 2640 raft_consensus.cc:482] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:06.405249 2640 raft_consensus.cc:504] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.406316 2640 leader_election.cc:288] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495), 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283)
I20260927 02:44:06.410468 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.406626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40243 (local address 127.0.98.1:34495)
0927 02:44:06.406797 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.406803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.407013 (+ 210us) server_negotiation.cc:407] Connection header received
0927 02:44:06.407087 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.407090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.407146 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.407241 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:06.407559 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.408183 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.409368 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.409899 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.409989 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.410041 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.410275 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.410332 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 02:44:06.410504 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.406546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34495 (local address 127.0.98.2:40243)
0927 02:44:06.406862 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.406874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.406987 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.407288 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.407294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.407318 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:06.407434 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.407441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.408292 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.408296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.409243 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.409252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.410022 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.410025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.410112 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.410138 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.410244 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.410310 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260927 02:44:06.410878 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.406672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49299 (local address 127.0.98.3:40283)
0927 02:44:06.406874 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.406880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.407266 (+ 386us) server_negotiation.cc:407] Connection header received
0927 02:44:06.407341 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.407344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.407388 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.407499 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:06.407859 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.408585 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.409743 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.410297 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.410392 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.410447 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.410679 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.410768 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 02:44:06.411044 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c163d779a64428aa1f41c4ebd5ae129" candidate_uuid: "11d0f27eb33847dfa56214c3cb911f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76b2f758ce54157887a14bdaef369b5" is_pre_election: true
I20260927 02:44:06.411203 2283 raft_consensus.cc:2454] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d0f27eb33847dfa56214c3cb911f40 in term 0.
I20260927 02:44:06.411312 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.406580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40283 (local address 127.0.98.2:49299)
0927 02:44:06.407129 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.407144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.407261 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.407541 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.407544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.407563 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:06.407692 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.407699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.408707 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.408710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.409619 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.409628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.410414 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.410418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.410520 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.410547 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.411036 (+ 489us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.411109 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":122,"threads_started":1}
I20260927 02:44:06.411612 2357 leader_election.cc:302] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, f76b2f758ce54157887a14bdaef369b5; no voters:
I20260927 02:44:06.411912 2640 raft_consensus.cc:2781] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:06.411898 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c163d779a64428aa1f41c4ebd5ae129" candidate_uuid: "11d0f27eb33847dfa56214c3cb911f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63706950adfc47b68ba99f311ca25fb8" is_pre_election: true
I20260927 02:44:06.411978 2640 raft_consensus.cc:482] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:06.412009 2640 raft_consensus.cc:3037] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:06.412043 2563 raft_consensus.cc:2454] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d0f27eb33847dfa56214c3cb911f40 in term 0.
I20260927 02:44:06.413033 2640 raft_consensus.cc:504] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.413371 2640 leader_election.cc:288] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [CANDIDATE]: Term 1 election: Requested vote from peers f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495), 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283)
I20260927 02:44:06.413722 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c163d779a64428aa1f41c4ebd5ae129" candidate_uuid: "11d0f27eb33847dfa56214c3cb911f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63706950adfc47b68ba99f311ca25fb8"
I20260927 02:44:06.413730 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c163d779a64428aa1f41c4ebd5ae129" candidate_uuid: "11d0f27eb33847dfa56214c3cb911f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76b2f758ce54157887a14bdaef369b5"
I20260927 02:44:06.413846 2563 raft_consensus.cc:3037] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:06.413857 2283 raft_consensus.cc:3037] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:06.414764 2283 raft_consensus.cc:2454] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d0f27eb33847dfa56214c3cb911f40 in term 1.
I20260927 02:44:06.414835 2563 raft_consensus.cc:2454] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d0f27eb33847dfa56214c3cb911f40 in term 1.
I20260927 02:44:06.415045 2357 leader_election.cc:302] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, f76b2f758ce54157887a14bdaef369b5; no voters:
I20260927 02:44:06.415194 2640 raft_consensus.cc:2781] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:06.415385 2640 raft_consensus.cc:686] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 LEADER]: Becoming Leader. State: Replica: 11d0f27eb33847dfa56214c3cb911f40, State: Running, Role: LEADER
I20260927 02:44:06.415640 2640 consensus_queue.cc:237] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:06.418028 2126 catalog_manager.cc:5495] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "11d0f27eb33847dfa56214c3cb911f40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:06.453044 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.449385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.0.98.2:42755)
0927 02:44:06.449516 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.449521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.449620 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:44:06.449820 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.449823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.449872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.450029 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:06.450234 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.450758 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.451605 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.451827 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.451983 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.452233 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.452673 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.452803 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.452843 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.452893 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:44:06.462430 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.459454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44231 (local address 127.0.98.1:34495)
0927 02:44:06.459588 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.459592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.459694 (+ 102us) server_negotiation.cc:407] Connection header received
0927 02:44:06.459787 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.459789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.459828 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.459959 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:06.460205 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.460693 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.461504 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.461714 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.461798 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.462081 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.462140 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.462194 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.462296 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.462338 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:44:06.462981 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.459339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34495 (local address 127.0.98.2:44231)
0927 02:44:06.459620 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.459632 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.459713 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.459979 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.459981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.459987 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:06.460081 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.460085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.460786 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.460788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.461382 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.461390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.461860 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.461863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.461929 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.461968 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.462217 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.462222 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.462272 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.462331 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
I20260927 02:44:06.468812 2283 raft_consensus.cc:1261] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Refusing update from remote peer 11d0f27eb33847dfa56214c3cb911f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:06.468823 2563 raft_consensus.cc:1261] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Refusing update from remote peer 11d0f27eb33847dfa56214c3cb911f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:06.469286 2687 consensus_queue.cc:1035] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:06.469481 2640 consensus_queue.cc:1035] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:06.485370 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.482211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57561 (local address 127.0.98.2:42755)
0927 02:44:06.482312 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:06.482319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:06.482351 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:44:06.482470 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:06.482476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:06.482518 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:06.482615 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:06.482825 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.483491 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.484357 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.484558 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.484679 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.484975 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.485035 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.485111 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:06.485212 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:44:06.485255 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:44:06.485392 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:06.482114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42755 (local address 127.0.98.1:57561)
0927 02:44:06.482285 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:06.482302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:06.482370 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:06.482637 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:06.482639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:06.482646 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:06.482721 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.482726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.483610 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.483613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.484263 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:06.484272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:06.484714 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:06.484720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:06.484806 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:06.484829 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:06.485126 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:06.485135 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 02:44:06.485203 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:44:06.485283 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 02:44:06.486092 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:44:08.040325 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:08.040468 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:08.042732 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: Sending DeleteTablet for 3 replicas of tablet 8c163d779a64428aa1f41c4ebd5ae129
I20260927 02:44:08.043300 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c163d779a64428aa1f41c4ebd5ae129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43364
I20260927 02:44:08.043394 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c163d779a64428aa1f41c4ebd5ae129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58114
I20260927 02:44:08.043608 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c163d779a64428aa1f41c4ebd5ae129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39730
I20260927 02:44:08.043766 2719 tablet_replica.cc:330] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: stopping tablet replica
I20260927 02:44:08.043947 2719 raft_consensus.cc:2229] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:08.043957 2718 tablet_replica.cc:330] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: stopping tablet replica
I20260927 02:44:08.044076 2718 raft_consensus.cc:2229] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:08.044204 2718 raft_consensus.cc:2258] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:08.044425 2719 raft_consensus.cc:2258] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:08.044701 2720 tablet_replica.cc:330] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: stopping tablet replica
I20260927 02:44:08.044836 2720 raft_consensus.cc:2229] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:08.045075 2720 raft_consensus.cc:2258] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:08.045163 2719 ts_tablet_manager.cc:1905] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:08.045641 2718 ts_tablet_manager.cc:1905] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:08.045800 2720 ts_tablet_manager.cc:1905] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:08.048053 2719 ts_tablet_manager.cc:1918] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:44:08.048146 2719 log.cc:1196] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/8c163d779a64428aa1f41c4ebd5ae129
I20260927 02:44:08.048238 2718 ts_tablet_manager.cc:1918] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:44:08.048326 2718 log.cc:1196] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/8c163d779a64428aa1f41c4ebd5ae129
I20260927 02:44:08.048461 2719 ts_tablet_manager.cc:1939] T 8c163d779a64428aa1f41c4ebd5ae129 P 11d0f27eb33847dfa56214c3cb911f40: Deleting consensus metadata
I20260927 02:44:08.048594 2718 ts_tablet_manager.cc:1939] T 8c163d779a64428aa1f41c4ebd5ae129 P f76b2f758ce54157887a14bdaef369b5: Deleting consensus metadata
I20260927 02:44:08.048595 2720 ts_tablet_manager.cc:1918] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:44:08.048743 2720 log.cc:1196] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/8c163d779a64428aa1f41c4ebd5ae129
I20260927 02:44:08.049043 2720 ts_tablet_manager.cc:1939] T 8c163d779a64428aa1f41c4ebd5ae129 P 63706950adfc47b68ba99f311ca25fb8: Deleting consensus metadata
I20260927 02:44:08.049126 2112 catalog_manager.cc:4841] TS 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755): tablet 8c163d779a64428aa1f41c4ebd5ae129 (table test-table [id=01df85342a5c458cb9eb581a81f876e3]) successfully deleted
I20260927 02:44:08.049368 2113 catalog_manager.cc:4841] TS f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495): tablet 8c163d779a64428aa1f41c4ebd5ae129 (table test-table [id=01df85342a5c458cb9eb581a81f876e3]) successfully deleted
I20260927 02:44:08.049784 2110 catalog_manager.cc:4841] TS 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283): tablet 8c163d779a64428aa1f41c4ebd5ae129 (table test-table [id=01df85342a5c458cb9eb581a81f876e3]) successfully deleted
Sep 27 02:44:08 dist-test-slave-gkrl krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:44:08 dist-test-slave-gkrl krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:08.072744 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:08.063135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.0.98.62:33503)
0927 02:44:08.063405 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:08.063412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:08.063472 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:44:08.063611 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:08.063613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:08.063667 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:08.063744 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:08.064545 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:08.065080 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:08.066293 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:08.066531 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:08.066624 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:08.069389 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:08.069392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:08.069394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:08.069415 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:08.070961 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:08.071396 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:08.071401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:08.071403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:08.071483 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:08.071764 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:08.071768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:08.071770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:08.072062 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:08.072200 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:08.072444 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 02:44:08.072570 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260927 02:44:08.075202 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:08.075649 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:08.080396 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2fb9c55574410590ad8ce74da93564 (DEFAULT_TABLE table=test-table [id=63074f5705ee46f7be19a0229cc85114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:08.080562 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2fb9c55574410590ad8ce74da93564 (DEFAULT_TABLE table=test-table [id=63074f5705ee46f7be19a0229cc85114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:08.080696 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2fb9c55574410590ad8ce74da93564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:08.080708 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2fb9c55574410590ad8ce74da93564 (DEFAULT_TABLE table=test-table [id=63074f5705ee46f7be19a0229cc85114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:08.080845 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2fb9c55574410590ad8ce74da93564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:08.080960 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2fb9c55574410590ad8ce74da93564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:08.083186 2729 tablet_bootstrap.cc:492] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: Bootstrap starting.
I20260927 02:44:08.083599 2731 tablet_bootstrap.cc:492] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: Bootstrap starting.
I20260927 02:44:08.083909 2730 tablet_bootstrap.cc:492] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: Bootstrap starting.
I20260927 02:44:08.084288 2729 tablet_bootstrap.cc:654] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:08.084391 2731 tablet_bootstrap.cc:654] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:08.084751 2730 tablet_bootstrap.cc:654] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:08.085493 2729 tablet_bootstrap.cc:492] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: No bootstrap required, opened a new log
I20260927 02:44:08.085589 2729 ts_tablet_manager.cc:1397] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:08.085790 2730 tablet_bootstrap.cc:492] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: No bootstrap required, opened a new log
I20260927 02:44:08.085873 2730 ts_tablet_manager.cc:1397] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:08.086015 2729 raft_consensus.cc:348] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.086127 2729 raft_consensus.cc:374] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:08.086169 2729 raft_consensus.cc:729] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d0f27eb33847dfa56214c3cb911f40, State: Initialized, Role: FOLLOWER
I20260927 02:44:08.086290 2729 consensus_queue.cc:260] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.086292 2730 raft_consensus.cc:348] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.086380 2730 raft_consensus.cc:374] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:08.086414 2730 raft_consensus.cc:729] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63706950adfc47b68ba99f311ca25fb8, State: Initialized, Role: FOLLOWER
I20260927 02:44:08.086532 2730 consensus_queue.cc:260] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.086580 2729 ts_tablet_manager.cc:1428] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:08.086894 2730 ts_tablet_manager.cc:1428] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:08.087080 2731 tablet_bootstrap.cc:492] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: No bootstrap required, opened a new log
I20260927 02:44:08.087180 2731 ts_tablet_manager.cc:1397] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 02:44:08.087612 2731 raft_consensus.cc:348] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.087764 2731 raft_consensus.cc:374] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:08.087805 2731 raft_consensus.cc:729] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76b2f758ce54157887a14bdaef369b5, State: Initialized, Role: FOLLOWER
I20260927 02:44:08.087985 2731 consensus_queue.cc:260] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.088256 2731 ts_tablet_manager.cc:1428] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:08.207042 2712 raft_consensus.cc:482] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:08.207180 2712 raft_consensus.cc:504] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.207532 2712 leader_election.cc:288] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495), 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283)
I20260927 02:44:08.208014 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2fb9c55574410590ad8ce74da93564" candidate_uuid: "11d0f27eb33847dfa56214c3cb911f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76b2f758ce54157887a14bdaef369b5" is_pre_election: true
I20260927 02:44:08.208014 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2fb9c55574410590ad8ce74da93564" candidate_uuid: "11d0f27eb33847dfa56214c3cb911f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63706950adfc47b68ba99f311ca25fb8" is_pre_election: true
I20260927 02:44:08.208176 2283 raft_consensus.cc:2454] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d0f27eb33847dfa56214c3cb911f40 in term 0.
I20260927 02:44:08.208177 2563 raft_consensus.cc:2454] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d0f27eb33847dfa56214c3cb911f40 in term 0.
I20260927 02:44:08.208550 2357 leader_election.cc:302] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, f76b2f758ce54157887a14bdaef369b5; no voters:
I20260927 02:44:08.208762 2712 raft_consensus.cc:2781] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:08.208823 2712 raft_consensus.cc:482] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:08.208855 2712 raft_consensus.cc:3037] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:08.209879 2712 raft_consensus.cc:504] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.210207 2712 leader_election.cc:288] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [CANDIDATE]: Term 1 election: Requested vote from peers f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495), 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283)
I20260927 02:44:08.210564 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2fb9c55574410590ad8ce74da93564" candidate_uuid: "11d0f27eb33847dfa56214c3cb911f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63706950adfc47b68ba99f311ca25fb8"
I20260927 02:44:08.210592 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2fb9c55574410590ad8ce74da93564" candidate_uuid: "11d0f27eb33847dfa56214c3cb911f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76b2f758ce54157887a14bdaef369b5"
I20260927 02:44:08.210669 2563 raft_consensus.cc:3037] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:08.210743 2283 raft_consensus.cc:3037] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:08.211622 2563 raft_consensus.cc:2454] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d0f27eb33847dfa56214c3cb911f40 in term 1.
I20260927 02:44:08.211628 2283 raft_consensus.cc:2454] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d0f27eb33847dfa56214c3cb911f40 in term 1.
I20260927 02:44:08.211916 2354 leader_election.cc:302] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, 63706950adfc47b68ba99f311ca25fb8; no voters:
I20260927 02:44:08.212095 2712 raft_consensus.cc:2781] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:08.212204 2712 raft_consensus.cc:686] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 LEADER]: Becoming Leader. State: Replica: 11d0f27eb33847dfa56214c3cb911f40, State: Running, Role: LEADER
I20260927 02:44:08.212342 2712 consensus_queue.cc:237] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.213956 2126 catalog_manager.cc:5495] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "11d0f27eb33847dfa56214c3cb911f40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:08.245007 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:08.241893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.0.98.2:42755)
0927 02:44:08.242198 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:08.242205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:08.242217 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:44:08.242254 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:08.242256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:08.242293 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:08.242365 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:08.242649 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:08.243165 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:08.243996 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:08.244233 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:08.244317 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:08.244561 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:08.244627 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:08.244684 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:08.244772 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 02:44:08.244817 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260927 02:44:08.251313 2283 raft_consensus.cc:1261] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Refusing update from remote peer 11d0f27eb33847dfa56214c3cb911f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:08.251336 2563 raft_consensus.cc:1261] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Refusing update from remote peer 11d0f27eb33847dfa56214c3cb911f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:08.251742 2712 consensus_queue.cc:1035] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:08.251921 2695 consensus_queue.cc:1035] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:08.282217 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:08.282362 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:08.284358 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: Sending DeleteTablet for 3 replicas of tablet 0f2fb9c55574410590ad8ce74da93564
I20260927 02:44:08.284907 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2fb9c55574410590ad8ce74da93564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43364
I20260927 02:44:08.284907 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2fb9c55574410590ad8ce74da93564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58114
I20260927 02:44:08.284953 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2fb9c55574410590ad8ce74da93564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39730
I20260927 02:44:08.285095 2718 tablet_replica.cc:330] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: stopping tablet replica
I20260927 02:44:08.285161 2719 tablet_replica.cc:330] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: stopping tablet replica
I20260927 02:44:08.285199 2720 tablet_replica.cc:330] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: stopping tablet replica
I20260927 02:44:08.285215 2718 raft_consensus.cc:2229] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:08.285271 2719 raft_consensus.cc:2229] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:08.285311 2718 raft_consensus.cc:2258] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:08.285305 2720 raft_consensus.cc:2229] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:08.285395 2720 raft_consensus.cc:2258] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:08.285445 2719 raft_consensus.cc:2258] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:08.285845 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:44:08.286110 2719 ts_tablet_manager.cc:1905] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:08.286193 2718 ts_tablet_manager.cc:1905] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:08.286228 2720 ts_tablet_manager.cc:1905] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:44:08.286337 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:08.288614 2720 ts_tablet_manager.cc:1918] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:08.288614 2718 ts_tablet_manager.cc:1918] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:08.288686 2718 log.cc:1196] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/0f2fb9c55574410590ad8ce74da93564
I20260927 02:44:08.288686 2720 log.cc:1196] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/0f2fb9c55574410590ad8ce74da93564
I20260927 02:44:08.288846 2719 ts_tablet_manager.cc:1918] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:08.288921 2719 log.cc:1196] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/0f2fb9c55574410590ad8ce74da93564
I20260927 02:44:08.288944 2718 ts_tablet_manager.cc:1939] T 0f2fb9c55574410590ad8ce74da93564 P f76b2f758ce54157887a14bdaef369b5: Deleting consensus metadata
I20260927 02:44:08.288944 2720 ts_tablet_manager.cc:1939] T 0f2fb9c55574410590ad8ce74da93564 P 63706950adfc47b68ba99f311ca25fb8: Deleting consensus metadata
I20260927 02:44:08.289194 2719 ts_tablet_manager.cc:1939] T 0f2fb9c55574410590ad8ce74da93564 P 11d0f27eb33847dfa56214c3cb911f40: Deleting consensus metadata
I20260927 02:44:08.289871 2112 catalog_manager.cc:4841] TS 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755): tablet 0f2fb9c55574410590ad8ce74da93564 (table test-table [id=63074f5705ee46f7be19a0229cc85114]) successfully deleted
I20260927 02:44:08.290325 2110 catalog_manager.cc:4841] TS 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283): tablet 0f2fb9c55574410590ad8ce74da93564 (table test-table [id=63074f5705ee46f7be19a0229cc85114]) successfully deleted
I20260927 02:44:08.291105 2113 catalog_manager.cc:4841] TS f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495): tablet 0f2fb9c55574410590ad8ce74da93564 (table test-table [id=63074f5705ee46f7be19a0229cc85114]) successfully deleted
I20260927 02:44:08.291227 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 884c098999bc4f99884dabff238be27a (DEFAULT_TABLE table=test-table [id=4a1e2cd1e8e64a6c83530548c015d0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:08.291379 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 884c098999bc4f99884dabff238be27a (DEFAULT_TABLE table=test-table [id=4a1e2cd1e8e64a6c83530548c015d0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:08.291427 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 884c098999bc4f99884dabff238be27a (DEFAULT_TABLE table=test-table [id=4a1e2cd1e8e64a6c83530548c015d0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:08.291577 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884c098999bc4f99884dabff238be27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:08.291632 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884c098999bc4f99884dabff238be27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:08.291704 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884c098999bc4f99884dabff238be27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:08.293715 2729 tablet_bootstrap.cc:492] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: Bootstrap starting.
I20260927 02:44:08.293987 2730 tablet_bootstrap.cc:492] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: Bootstrap starting.
I20260927 02:44:08.294596 2729 tablet_bootstrap.cc:654] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:08.294831 2730 tablet_bootstrap.cc:654] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:08.296114 2729 tablet_bootstrap.cc:492] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: No bootstrap required, opened a new log
I20260927 02:44:08.296188 2729 ts_tablet_manager.cc:1397] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:08.296617 2730 tablet_bootstrap.cc:492] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: No bootstrap required, opened a new log
I20260927 02:44:08.296645 2729 raft_consensus.cc:348] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.296747 2730 ts_tablet_manager.cc:1397] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:08.296769 2729 raft_consensus.cc:374] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:08.296815 2729 raft_consensus.cc:729] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d0f27eb33847dfa56214c3cb911f40, State: Initialized, Role: FOLLOWER
I20260927 02:44:08.296945 2729 consensus_queue.cc:260] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.297242 2729 ts_tablet_manager.cc:1428] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:08.297194 2730 raft_consensus.cc:348] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.297304 2730 raft_consensus.cc:374] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:08.297354 2730 raft_consensus.cc:729] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63706950adfc47b68ba99f311ca25fb8, State: Initialized, Role: FOLLOWER
I20260927 02:44:08.297487 2730 consensus_queue.cc:260] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.297772 2730 ts_tablet_manager.cc:1428] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:08.298290 2731 tablet_bootstrap.cc:492] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5: Bootstrap starting.
I20260927 02:44:08.299108 2731 tablet_bootstrap.cc:654] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:08.300622 2731 tablet_bootstrap.cc:492] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5: No bootstrap required, opened a new log
I20260927 02:44:08.300696 2731 ts_tablet_manager.cc:1397] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:08.301057 2731 raft_consensus.cc:348] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.301151 2731 raft_consensus.cc:374] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:08.301209 2731 raft_consensus.cc:729] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76b2f758ce54157887a14bdaef369b5, State: Initialized, Role: FOLLOWER
I20260927 02:44:08.301347 2731 consensus_queue.cc:260] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.301635 2731 ts_tablet_manager.cc:1428] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:08.384940 2735 raft_consensus.cc:482] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:08.385100 2735 raft_consensus.cc:504] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.385581 2735 leader_election.cc:288] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495), 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755)
I20260927 02:44:08.386047 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884c098999bc4f99884dabff238be27a" candidate_uuid: "63706950adfc47b68ba99f311ca25fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76b2f758ce54157887a14bdaef369b5" is_pre_election: true
I20260927 02:44:08.386049 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884c098999bc4f99884dabff238be27a" candidate_uuid: "63706950adfc47b68ba99f311ca25fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d0f27eb33847dfa56214c3cb911f40" is_pre_election: true
I20260927 02:44:08.386209 2423 raft_consensus.cc:2454] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63706950adfc47b68ba99f311ca25fb8 in term 0.
I20260927 02:44:08.386209 2283 raft_consensus.cc:2454] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63706950adfc47b68ba99f311ca25fb8 in term 0.
I20260927 02:44:08.386561 2496 leader_election.cc:302] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, 63706950adfc47b68ba99f311ca25fb8, f76b2f758ce54157887a14bdaef369b5; no voters:
I20260927 02:44:08.386731 2735 raft_consensus.cc:2781] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:08.386792 2735 raft_consensus.cc:482] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:08.386837 2735 raft_consensus.cc:3037] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:08.388029 2735 raft_consensus.cc:504] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.388337 2735 leader_election.cc:288] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [CANDIDATE]: Term 1 election: Requested vote from peers f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495), 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755)
I20260927 02:44:08.388695 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884c098999bc4f99884dabff238be27a" candidate_uuid: "63706950adfc47b68ba99f311ca25fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d0f27eb33847dfa56214c3cb911f40"
I20260927 02:44:08.388695 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884c098999bc4f99884dabff238be27a" candidate_uuid: "63706950adfc47b68ba99f311ca25fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76b2f758ce54157887a14bdaef369b5"
I20260927 02:44:08.388823 2283 raft_consensus.cc:3037] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:08.388823 2423 raft_consensus.cc:3037] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:08.389925 2283 raft_consensus.cc:2454] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63706950adfc47b68ba99f311ca25fb8 in term 1.
I20260927 02:44:08.389918 2423 raft_consensus.cc:2454] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63706950adfc47b68ba99f311ca25fb8 in term 1.
I20260927 02:44:08.390249 2496 leader_election.cc:302] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f27eb33847dfa56214c3cb911f40, 63706950adfc47b68ba99f311ca25fb8; no voters:
I20260927 02:44:08.390414 2735 raft_consensus.cc:2781] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:08.390604 2735 raft_consensus.cc:686] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 63706950adfc47b68ba99f311ca25fb8, State: Running, Role: LEADER
I20260927 02:44:08.390767 2735 consensus_queue.cc:237] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } }
I20260927 02:44:08.392587 2126 catalog_manager.cc:5495] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "63706950adfc47b68ba99f311ca25fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63706950adfc47b68ba99f311ca25fb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40283 } health_report { overall_health: HEALTHY } } }
I20260927 02:44:08.428977 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:08.425803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.0.98.3:40283)
0927 02:44:08.426088 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:08.426095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:08.426107 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:44:08.426148 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:08.426150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:08.426188 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:08.426258 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:08.426528 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:08.427043 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:08.427947 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:08.428191 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:08.428271 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:08.428533 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:08.428602 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:08.428667 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:08.428738 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:44:08.428782 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":146,"threads_started":1}
I20260927 02:44:08.434638 2283 raft_consensus.cc:1261] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Refusing update from remote peer 63706950adfc47b68ba99f311ca25fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:08.434803 2423 raft_consensus.cc:1261] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Refusing update from remote peer 63706950adfc47b68ba99f311ca25fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:08.435083 2735 consensus_queue.cc:1035] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76b2f758ce54157887a14bdaef369b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:08.435456 2735 consensus_queue.cc:1035] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d0f27eb33847dfa56214c3cb911f40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:44:08.447816 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:44:09.996014 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:09.996160 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:09.998446 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e97e5351a6e4735a0732372ba8d9e75: Sending DeleteTablet for 3 replicas of tablet 884c098999bc4f99884dabff238be27a
I20260927 02:44:09.998911 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 884c098999bc4f99884dabff238be27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43364
I20260927 02:44:09.999082 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 884c098999bc4f99884dabff238be27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39730
I20260927 02:44:09.999195 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 884c098999bc4f99884dabff238be27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58114
I20260927 02:44:09.999243 2774 tablet_replica.cc:330] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5: stopping tablet replica
I20260927 02:44:09.999378 2774 raft_consensus.cc:2229] T 884c098999bc4f99884dabff238be27a P f76b2f758ce54157887a14bdaef369b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:09.999411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 2198
I20260927 02:44:10.000932 2776 tablet_replica.cc:330] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: stopping tablet replica
I20260927 02:44:10.001045 2776 raft_consensus.cc:2229] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:10.001134 2776 raft_consensus.cc:2258] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:10.001332 2775 tablet_replica.cc:330] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: stopping tablet replica
I20260927 02:44:10.001448 2775 raft_consensus.cc:2229] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:10.001611 2775 raft_consensus.cc:2258] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:10.002084 2776 ts_tablet_manager.cc:1905] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:10.002235 2775 ts_tablet_manager.cc:1905] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:10.005159 2776 ts_tablet_manager.cc:1918] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:44:10.005241 2776 log.cc:1196] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/884c098999bc4f99884dabff238be27a
I20260927 02:44:10.005520 2776 ts_tablet_manager.cc:1939] T 884c098999bc4f99884dabff238be27a P 11d0f27eb33847dfa56214c3cb911f40: Deleting consensus metadata
I20260927 02:44:10.005573 2775 ts_tablet_manager.cc:1918] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:44:10.005661 2775 log.cc:1196] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/884c098999bc4f99884dabff238be27a
I20260927 02:44:10.005913 2775 ts_tablet_manager.cc:1939] T 884c098999bc4f99884dabff238be27a P 63706950adfc47b68ba99f311ca25fb8: Deleting consensus metadata
I20260927 02:44:10.006225 2112 catalog_manager.cc:4841] TS 11d0f27eb33847dfa56214c3cb911f40 (127.0.98.2:42755): tablet 884c098999bc4f99884dabff238be27a (table test-table [id=4a1e2cd1e8e64a6c83530548c015d0e6]) successfully deleted
I20260927 02:44:10.006582 2110 catalog_manager.cc:4841] TS 63706950adfc47b68ba99f311ca25fb8 (127.0.98.3:40283): tablet 884c098999bc4f99884dabff238be27a (table test-table [id=4a1e2cd1e8e64a6c83530548c015d0e6]) successfully deleted
W20260927 02:44:10.009066 2113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34495 (error 108)
I20260927 02:44:10.009239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 2338
I20260927 02:44:10.012221 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:10.011684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34495 (local address 127.0.0.1:43380)
0927 02:44:10.011992 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:10.012083 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260927 02:44:10.012338 2113 catalog_manager.cc:4584] TS f76b2f758ce54157887a14bdaef369b5 (127.0.98.1:34495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 884c098999bc4f99884dabff238be27a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34495: connect: Connection refused (error 111)
I20260927 02:44:10.017977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 2478
I20260927 02:44:10.025158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 2094
2026-09-27T02:44:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6294 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:10 dist-test-slave-gkrl krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:10 dist-test-slave-gkrl krb5kdc[2781](info): set up 2 sockets
Sep 27 02:44:10 dist-test-slave-gkrl krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260927 02:44:11.087898 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:11 dist-test-slave-gkrl krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:11.207406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39283
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:11.314181 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:11.314463 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:11.314534 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:11.318594 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:11.318682 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:11.318722 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:11.318780 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:11.318817 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:11.324054 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:11.325078 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:11.326093 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:11.334868 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:11.334916 2794 server_base.cc:1034] running on GCE node
W20260927 02:44:11.334856 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:11.334846 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:11.335551 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:11.336627 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:11.337790 2794 hybrid_clock.cc:648] HybridClock initialized: now 1790477051337760 us; error 47 us; skew 500 ppm
Sep 27 02:44:11 dist-test-slave-gkrl krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:11.340557 2794 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:11.341411 2794 webserver.cc:460] Webserver started at http://127.0.98.62:40475/ using document root <none> and password file <none>
I20260927 02:44:11.341964 2794 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:11.342049 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:11.342265 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:11.344245 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "56f0b5c4307d8927395f3b44490aee28"
server_key_iv: "bff5ae11898263655cfb2b771c474a8f"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.344833 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "56f0b5c4307d8927395f3b44490aee28"
server_key_iv: "bff5ae11898263655cfb2b771c474a8f"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.348425 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:11.350757 2809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:11.352044 2794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:11.352170 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "56f0b5c4307d8927395f3b44490aee28"
server_key_iv: "bff5ae11898263655cfb2b771c474a8f"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.352272 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:11.364243 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:11.366926 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:11.367122 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:11.374068 2794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39283
I20260927 02:44:11.374109 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39283 every 8 connection(s)
I20260927 02:44:11.374953 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:11.377622 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:11.382705 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea: Bootstrap starting.
I20260927 02:44:11.382838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 2794
I20260927 02:44:11.382941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:11.383189 392 external_mini_cluster.cc:1428] Setting key 7cda9fee1a57a30d1375116e6320c402
I20260927 02:44:11.385243 2862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:11.385926 2862 log.cc:824] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:11.387732 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea: No bootstrap required, opened a new log
I20260927 02:44:11.391105 2862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } }
I20260927 02:44:11.391325 2862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:11.391405 2862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787dc23ad6a74e4fab2d4fe8eff0f3ea, State: Initialized, Role: FOLLOWER
Sep 27 02:44:11 dist-test-slave-gkrl krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:11.391857 2862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } }
I20260927 02:44:11.392051 2862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:11.392133 2862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:11.392232 2862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:11.393231 2862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } }
I20260927 02:44:11.393579 2862 leader_election.cc:302] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787dc23ad6a74e4fab2d4fe8eff0f3ea; no voters:
I20260927 02:44:11.393877 2862 leader_election.cc:288] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:11.393980 2867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:11.394189 2867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [term 1 LEADER]: Becoming Leader. State: Replica: 787dc23ad6a74e4fab2d4fe8eff0f3ea, State: Running, Role: LEADER
I20260927 02:44:11.394556 2867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } }
I20260927 02:44:11.395169 2862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:11.397722 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } } }
I20260927 02:44:11.397866 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:11.398011 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:11.384482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.0.98.62:39283)
0927 02:44:11.384884 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:11.384897 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:11.384929 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:44:11.385573 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:11.385591 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:11.385936 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:11.386298 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:11.387112 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.388139 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.388985 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.389459 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.389630 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:11.392350 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:11.392359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:11.392374 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:11.392403 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:11.394365 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:11.394860 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:11.394865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:11.394870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:11.394999 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:11.395325 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:11.395328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:11.395330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:11.395799 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:11.396613 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:11.396791 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 02:44:11.397090 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260927 02:44:11.398123 2869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787dc23ad6a74e4fab2d4fe8eff0f3ea. Latest consensus state: current_term: 1 leader_uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787dc23ad6a74e4fab2d4fe8eff0f3ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } } }
I20260927 02:44:11.398196 2869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea [sys.catalog]: This master's current role is: LEADER
W20260927 02:44:11.400286 2882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:44:11.400367 2882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:44:11.400458 2883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:11.401141 2883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:11.405735 2883 catalog_manager.cc:1347] Generated new cluster ID: ef4b39376e2f4915bc612f344c568d0d
I20260927 02:44:11.405794 2883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:11.417351 2883 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:11.418187 2883 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:11.424139 2883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787dc23ad6a74e4fab2d4fe8eff0f3ea: Generated new TSK 0
I20260927 02:44:11.424862 2883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:11.463045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:11.574360 2888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:11.574638 2888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:11.574716 2888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:11.578670 2888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:11.578755 2888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:11.578841 2888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:11.583835 2888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:11.584797 2888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:11.585752 2888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:11.595175 2893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:11.595182 2894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:11.595197 2896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:11.595388 2888 server_base.cc:1034] running on GCE node
I20260927 02:44:11.595860 2888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:11.596594 2888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:11.597790 2888 hybrid_clock.cc:648] HybridClock initialized: now 1790477051597772 us; error 37 us; skew 500 ppm
Sep 27 02:44:11 dist-test-slave-gkrl krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:11.600503 2888 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:11.601344 2888 webserver.cc:460] Webserver started at http://127.0.98.1:40771/ using document root <none> and password file <none>
I20260927 02:44:11.601882 2888 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:11.602020 2888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:11.602392 2888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:11.604665 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "77912923fbd04a438e2ac1e250d84f32"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "52b0e8f532bd1274f54e9ee1c8928044"
server_key_iv: "3d1cddf8ad0056f43b542e843a15475d"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.605248 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77912923fbd04a438e2ac1e250d84f32"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "52b0e8f532bd1274f54e9ee1c8928044"
server_key_iv: "3d1cddf8ad0056f43b542e843a15475d"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.608808 2888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:44:11.611016 2903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:11.612275 2888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:11.612406 2888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "77912923fbd04a438e2ac1e250d84f32"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "52b0e8f532bd1274f54e9ee1c8928044"
server_key_iv: "3d1cddf8ad0056f43b542e843a15475d"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.612509 2888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:11.625183 2888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:11.627952 2888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:11.628142 2888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:11.628679 2888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:11.629482 2888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:11.629560 2888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:11.629628 2888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:11.629680 2888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:11.638643 2888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34773
I20260927 02:44:11.638674 3016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34773 every 8 connection(s)
I20260927 02:44:11.639545 2888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:44:11 dist-test-slave-gkrl krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:11.649319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 2888
I20260927 02:44:11.649439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:11.649722 392 external_mini_cluster.cc:1428] Setting key 789ac2df1897385edf64b4cbe2b8aa6e
I20260927 02:44:11.653820 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:11.641329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43315 (local address 127.0.98.62:39283)
0927 02:44:11.641462 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:11.641466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:11.642292 (+ 826us) server_negotiation.cc:407] Connection header received
0927 02:44:11.643685 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:11.643688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:11.643741 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:11.643846 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:11.644976 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.645505 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.646613 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.646856 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.646968 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:11.649564 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:11.649567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:11.649569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:11.649592 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:11.651330 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:11.652015 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:11.652019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:11.652021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:11.652108 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:11.652521 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:11.652523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:11.652524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:11.652707 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:11.652790 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:11.653442 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 02:44:11.653596 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:44:11.654359 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:11.641716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39283 (local address 127.0.98.1:43315)
0927 02:44:11.642152 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:11.642193 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:11.643484 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:11.644049 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:11.644056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:11.644397 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:11.644834 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:11.644847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.645632 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.645636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:11.646487 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:11.646496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.647000 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.647004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:11.647169 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:11.647718 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:11.647735 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:11.649351 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:11.651463 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:11.651468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:11.651482 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:11.651828 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:11.652214 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:11.652220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:11.652225 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:11.652410 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:11.652811 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:11.652819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:11.653133 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 02:44:11.653328 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 02:44:11.653574 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260927 02:44:11.656217 3017 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39283
I20260927 02:44:11.656561 3017 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:11.657099 3017 heartbeater.cc:511] Master 127.0.98.62:39283 requested a full tablet report, sending...
I20260927 02:44:11.658852 2826 ts_manager.cc:194] Registered new tserver with Master: 77912923fbd04a438e2ac1e250d84f32 (127.0.98.1:34773)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:44:11.660602 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:11.680034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:11.790089 3022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:11.790323 3022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:11.790364 3022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:11.794508 3022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:11.794564 3022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:11.794618 3022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:11.800010 3022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:11.800886 3022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:11.801820 3022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:11.811103 3030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:11.811122 3027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:11.811240 3022 server_base.cc:1034] running on GCE node
W20260927 02:44:11.811084 3028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:11.811690 3022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:11.812337 3022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:11.813547 3022 hybrid_clock.cc:648] HybridClock initialized: now 1790477051813521 us; error 25 us; skew 500 ppm
Sep 27 02:44:11 dist-test-slave-gkrl krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:11.816548 3022 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:11.817374 3022 webserver.cc:460] Webserver started at http://127.0.98.2:33277/ using document root <none> and password file <none>
I20260927 02:44:11.817917 3022 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:11.818029 3022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:11.818259 3022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:11.820227 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "d5f3b992268e4b29a3d030d5d6607321"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "9a61f26ca51a039b860e3e788c1c7e6d"
server_key_iv: "ee168e2d87c906de6b4da7baf6f6b13f"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.820772 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d5f3b992268e4b29a3d030d5d6607321"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "9a61f26ca51a039b860e3e788c1c7e6d"
server_key_iv: "ee168e2d87c906de6b4da7baf6f6b13f"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.824391 3022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:11.826783 3037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:11.828105 3022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:11.828241 3022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "d5f3b992268e4b29a3d030d5d6607321"
format_stamp: "Formatted at 2026-09-27 02:44:11 on dist-test-slave-gkrl"
server_key: "9a61f26ca51a039b860e3e788c1c7e6d"
server_key_iv: "ee168e2d87c906de6b4da7baf6f6b13f"
server_key_version: "encryptionkey@0"
I20260927 02:44:11.828351 3022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:11.850340 3022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:11.853448 3022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:11.853653 3022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:11.854235 3022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:11.855094 3022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:11.855159 3022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:11.855224 3022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:11.855270 3022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:11.863852 3022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46703
I20260927 02:44:11.863929 3150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46703 every 8 connection(s)
I20260927 02:44:11.864795 3022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:11.866003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 3022
I20260927 02:44:11.866113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:11.866389 392 external_mini_cluster.cc:1428] Setting key b04bd8468f3029b1ac241452a6365447
Sep 27 02:44:11 dist-test-slave-gkrl krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:44:11.880251 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:11.866668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43455 (local address 127.0.98.62:39283)
0927 02:44:11.866837 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:11.866841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:11.867826 (+ 985us) server_negotiation.cc:407] Connection header received
0927 02:44:11.869931 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:11.869934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:11.869982 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:11.870060 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:11.871155 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.871741 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.873544 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.873760 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.873842 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:11.876650 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:11.876653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:11.876655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:11.876676 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:11.878167 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:11.878706 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:11.878708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:11.878710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:11.878792 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:11.879151 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:11.879155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:11.879157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:11.879331 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:11.879439 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:11.879956 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 02:44:11.880083 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:44:11.881076 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:11.867162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39283 (local address 127.0.98.2:43455)
0927 02:44:11.867675 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:11.867732 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:11.869691 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:11.870227 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:11.870234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:11.870567 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:11.871015 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:11.871028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.871860 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.871863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:11.873393 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:11.873409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:11.873917 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:11.873925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:11.874141 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:11.874786 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:11.874810 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:11.876463 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:11.878286 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:11.878292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:11.878308 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:11.878571 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:11.878886 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:11.878890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:11.878893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:11.879039 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:11.879437 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:11.879444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:11.879686 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 02:44:11.879866 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 02:44:11.880204 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":133,"threads_started":1}
I20260927 02:44:11.882210 3151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39283
I20260927 02:44:11.882473 3151 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:11.883055 3151 heartbeater.cc:511] Master 127.0.98.62:39283 requested a full tablet report, sending...
I20260927 02:44:11.884239 2826 ts_manager.cc:194] Registered new tserver with Master: d5f3b992268e4b29a3d030d5d6607321 (127.0.98.2:46703)
I20260927 02:44:11.884811 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:11.895991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:12.002774 3156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:12.003122 3156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:12.003196 3156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:12.009627 3156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:12.009707 3156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:12.009802 3156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:12.014968 3156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:12.016122 3156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:12.017210 3156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:12.026611 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:12.026603 3161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:12.026603 3162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:12.026692 3156 server_base.cc:1034] running on GCE node
I20260927 02:44:12.027226 3156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:12.027853 3156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:12.029110 3156 hybrid_clock.cc:648] HybridClock initialized: now 1790477052029099 us; error 29 us; skew 500 ppm
Sep 27 02:44:12 dist-test-slave-gkrl krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:12.032462 3156 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:12.033347 3156 webserver.cc:460] Webserver started at http://127.0.98.3:37199/ using document root <none> and password file <none>
I20260927 02:44:12.033892 3156 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:12.033988 3156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:12.034193 3156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:12.036202 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "796ab452add64fcd88949cb23fdf2c2a"
format_stamp: "Formatted at 2026-09-27 02:44:12 on dist-test-slave-gkrl"
server_key: "3cf7baf9353f291b1219da8f1a0e17a1"
server_key_iv: "6a272d5cd7eea432ae4c0881792b6804"
server_key_version: "encryptionkey@0"
I20260927 02:44:12.036744 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "796ab452add64fcd88949cb23fdf2c2a"
format_stamp: "Formatted at 2026-09-27 02:44:12 on dist-test-slave-gkrl"
server_key: "3cf7baf9353f291b1219da8f1a0e17a1"
server_key_iv: "6a272d5cd7eea432ae4c0881792b6804"
server_key_version: "encryptionkey@0"
I20260927 02:44:12.040292 3156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:12.042600 3171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:12.043776 3156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:12.043932 3156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "796ab452add64fcd88949cb23fdf2c2a"
format_stamp: "Formatted at 2026-09-27 02:44:12 on dist-test-slave-gkrl"
server_key: "3cf7baf9353f291b1219da8f1a0e17a1"
server_key_iv: "6a272d5cd7eea432ae4c0881792b6804"
server_key_version: "encryptionkey@0"
I20260927 02:44:12.044077 3156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:12.070652 3156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:12.073753 3156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:12.073949 3156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:12.074489 3156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:12.075258 3156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:12.075325 3156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:12.075394 3156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:12.075444 3156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:12.084635 3156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40299
I20260927 02:44:12.084679 3284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40299 every 8 connection(s)
I20260927 02:44:12.085489 3156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:12.092206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 3156
I20260927 02:44:12.092309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:12.092567 392 external_mini_cluster.cc:1428] Setting key 16dd90d31f1503313833f0a530243d8b
Sep 27 02:44:12 dist-test-slave-gkrl krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:12.099869 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:12.087264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53151 (local address 127.0.98.62:39283)
0927 02:44:12.087391 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:12.087396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:12.088228 (+ 832us) server_negotiation.cc:407] Connection header received
0927 02:44:12.089656 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:12.089659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:12.089706 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:12.089784 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:12.090871 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:12.091386 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:12.092993 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:12.093224 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:12.093335 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:12.096232 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:12.096235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:12.096237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:12.096261 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:12.098079 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:12.098543 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:12.098546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:12.098547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:12.098623 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:12.098921 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:12.098923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:12.098924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:12.099131 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:12.099217 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:12.099650 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 02:44:12.099746 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:44:12.100649 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:12.087551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39283 (local address 127.0.98.3:53151)
0927 02:44:12.088047 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:12.088121 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:12.089450 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:12.089932 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:12.089939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:12.090297 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:12.090738 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:12.090749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:12.091526 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:12.091529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:12.092833 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:12.092846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:12.093367 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:12.093371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:12.093602 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:12.094186 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:12.094207 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:12.096000 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:12.098185 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:12.098190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:12.098201 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:12.098435 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:12.098697 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:12.098699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:12.098700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:12.098820 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:12.099223 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:12.099228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:12.099429 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 02:44:12.099585 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 02:44:12.099808 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
I20260927 02:44:12.101776 3285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39283
I20260927 02:44:12.101999 3285 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:12.102406 3285 heartbeater.cc:511] Master 127.0.98.62:39283 requested a full tablet report, sending...
I20260927 02:44:12.103505 2826 ts_manager.cc:194] Registered new tserver with Master: 796ab452add64fcd88949cb23fdf2c2a (127.0.98.3:40299)
I20260927 02:44:12.104091 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53151
I20260927 02:44:12.106483 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:44:12.118230 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:12.116979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.0.98.62:39283)
0927 02:44:12.117112 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:12.117115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:12.117286 (+ 171us) server_negotiation.cc:407] Connection header received
0927 02:44:12.117389 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:12.117392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:12.117430 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:12.117502 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:12.118132 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36186: BlockingRecv error: recv got EOF from 127.0.0.1:36186 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:44:12.119400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 2888
I20260927 02:44:12.124948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 3022
I20260927 02:44:12.130709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 3156
I20260927 02:44:12.135661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 2794
2026-09-27T02:44:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2108 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:12 dist-test-slave-gkrl krb5kdc[3297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:12 dist-test-slave-gkrl krb5kdc[3297](info): set up 2 sockets
Sep 27 02:44:12 dist-test-slave-gkrl krb5kdc[3297](info): commencing operation
krb5kdc: starting...
W20260927 02:44:13.167831 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:13 dist-test-slave-gkrl krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:13.283165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:13.389509 3310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:13.389802 3310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:13.389873 3310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:13.393996 3310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:13.394073 3310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:13.394107 3310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:13.394150 3310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:13.394205 3310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:13.399199 3310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:13.400169 3310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:13.401118 3310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:13.409231 3315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:13.409276 3316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:13.409255 3318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:13.409591 3310 server_base.cc:1034] running on GCE node
I20260927 02:44:13.410080 3310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:13.411113 3310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:13.412307 3310 hybrid_clock.cc:648] HybridClock initialized: now 1790477053412287 us; error 30 us; skew 500 ppm
Sep 27 02:44:13 dist-test-slave-gkrl krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:13.415129 3310 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:13.415990 3310 webserver.cc:460] Webserver started at http://127.0.98.62:35681/ using document root <none> and password file <none>
I20260927 02:44:13.416534 3310 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:13.416622 3310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:13.416841 3310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:13.418838 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "e527c795894a44bda9eb11e4ab1198e2"
format_stamp: "Formatted at 2026-09-27 02:44:13 on dist-test-slave-gkrl"
server_key: "edfd1ef42ea2e1ad51cfef7e8013613d"
server_key_iv: "c302240c9e25015d3867785b5f09717f"
server_key_version: "encryptionkey@0"
I20260927 02:44:13.419370 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "e527c795894a44bda9eb11e4ab1198e2"
format_stamp: "Formatted at 2026-09-27 02:44:13 on dist-test-slave-gkrl"
server_key: "edfd1ef42ea2e1ad51cfef7e8013613d"
server_key_iv: "c302240c9e25015d3867785b5f09717f"
server_key_version: "encryptionkey@0"
I20260927 02:44:13.422820 3310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 02:44:13.425274 3325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:13.426519 3310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:13.426676 3310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "e527c795894a44bda9eb11e4ab1198e2"
format_stamp: "Formatted at 2026-09-27 02:44:13 on dist-test-slave-gkrl"
server_key: "edfd1ef42ea2e1ad51cfef7e8013613d"
server_key_iv: "c302240c9e25015d3867785b5f09717f"
server_key_version: "encryptionkey@0"
I20260927 02:44:13.426784 3310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:13.439546 3310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:13.442404 3310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:13.442588 3310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:13.449069 3310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42353
I20260927 02:44:13.449327 3377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42353 every 8 connection(s)
I20260927 02:44:13.450001 3310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:13.452607 3378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:13.457609 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2: Bootstrap starting.
I20260927 02:44:13.459028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 3310
I20260927 02:44:13.459187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:13.459444 392 external_mini_cluster.cc:1428] Setting key c7d734de0488cb877be5c554aa394b17
I20260927 02:44:13.460021 3378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:13.460855 3378 log.cc:824] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:13.462635 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2: No bootstrap required, opened a new log
I20260927 02:44:13.465899 3378 raft_consensus.cc:348] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527c795894a44bda9eb11e4ab1198e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } }
I20260927 02:44:13.466132 3378 raft_consensus.cc:374] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:13.466190 3378 raft_consensus.cc:729] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e527c795894a44bda9eb11e4ab1198e2, State: Initialized, Role: FOLLOWER
I20260927 02:44:13.466578 3378 consensus_queue.cc:260] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527c795894a44bda9eb11e4ab1198e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } }
I20260927 02:44:13.466698 3378 raft_consensus.cc:388] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:13.466784 3378 raft_consensus.cc:482] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:13.466882 3378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:44:13 dist-test-slave-gkrl krb5kdc[3297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:13.468020 3378 raft_consensus.cc:504] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527c795894a44bda9eb11e4ab1198e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } }
I20260927 02:44:13.468391 3378 leader_election.cc:302] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e527c795894a44bda9eb11e4ab1198e2; no voters:
I20260927 02:44:13.468690 3378 leader_election.cc:288] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:13.468778 3383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:13.469027 3383 raft_consensus.cc:686] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [term 1 LEADER]: Becoming Leader. State: Replica: e527c795894a44bda9eb11e4ab1198e2, State: Running, Role: LEADER
I20260927 02:44:13.469344 3383 consensus_queue.cc:237] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527c795894a44bda9eb11e4ab1198e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } }
I20260927 02:44:13.469903 3378 sys_catalog.cc:564] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:13.470485 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e527c795894a44bda9eb11e4ab1198e2. Latest consensus state: current_term: 1 leader_uuid: "e527c795894a44bda9eb11e4ab1198e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527c795894a44bda9eb11e4ab1198e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } }
I20260927 02:44:13.470681 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:13.471109 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:13.472718 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e527c795894a44bda9eb11e4ab1198e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527c795894a44bda9eb11e4ab1198e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } }
I20260927 02:44:13.472971 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:13.474093 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:13.474128 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:13.460416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.0.98.62:42353)
0927 02:44:13.460830 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:13.460843 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:13.460875 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:44:13.461694 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:13.461712 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:13.462001 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:13.462325 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:13.463134 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:13.464137 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:13.464978 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:13.465433 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:13.465600 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:13.468495 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:13.468500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:13.468514 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:13.468565 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:13.470568 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:13.471093 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:13.471100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:13.471108 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:13.471239 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:13.471528 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:13.471531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:13.471533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:13.472005 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:13.472211 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:13.472986 (+ 775us) server_negotiation.cc:299] Negotiation successful
0927 02:44:13.473266 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260927 02:44:13.477074 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:44:13.480182 3392 catalog_manager.cc:1347] Generated new cluster ID: 058a4ae05bae44e699b81e31bf77b0d1
I20260927 02:44:13.480239 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:13.493170 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:13.488449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.0.98.62:42353)
0927 02:44:13.488551 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:13.488555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:13.488860 (+ 305us) server_negotiation.cc:407] Connection header received
0927 02:44:13.489014 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:13.489017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:13.489065 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:13.489149 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:13.489632 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:13.490138 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:13.490957 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:13.491204 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:13.491313 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:13.492321 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:13.492325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:13.492327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:13.492737 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:13.492761 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:13.492776 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:13.492873 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:13.492965 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:44:13.493026 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:44:13.501096 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:13.502213 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:13.507501 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e527c795894a44bda9eb11e4ab1198e2: Generated new TSK 0
I20260927 02:44:13.508200 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:13.517963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 3310
2026-09-27T02:44:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1379 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:13 dist-test-slave-gkrl krb5kdc[3411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:13 dist-test-slave-gkrl krb5kdc[3411](info): set up 2 sockets
Sep 27 02:44:13 dist-test-slave-gkrl krb5kdc[3411](info): commencing operation
krb5kdc: starting...
W20260927 02:44:14.551924 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:14 dist-test-slave-gkrl krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:14.670862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:14.775579 3424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:14.775840 3424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:14.775941 3424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:14.779915 3424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:14.779970 3424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:14.779989 3424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:14.780005 3424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:14.780021 3424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:14.785172 3424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:14.786144 3424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:14.787117 3424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:14.795492 3429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:14.795574 3430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:14.795492 3432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:14.795836 3424 server_base.cc:1034] running on GCE node
I20260927 02:44:14.796451 3424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:14.797549 3424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:14.798763 3424 hybrid_clock.cc:648] HybridClock initialized: now 1790477054798736 us; error 51 us; skew 500 ppm
Sep 27 02:44:14 dist-test-slave-gkrl krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:14.801463 3424 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:14.802314 3424 webserver.cc:460] Webserver started at http://127.0.98.62:46233/ using document root <none> and password file <none>
I20260927 02:44:14.802809 3424 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:14.802866 3424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:14.803027 3424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:14.805022 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "b0c7bad3b7834c27839505b991095080"
format_stamp: "Formatted at 2026-09-27 02:44:14 on dist-test-slave-gkrl"
server_key: "75285e3024f51b8b7ab75933db8f0933"
server_key_iv: "681a7b98c47966446d75b0a11d9ab76f"
server_key_version: "encryptionkey@0"
I20260927 02:44:14.805585 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0c7bad3b7834c27839505b991095080"
format_stamp: "Formatted at 2026-09-27 02:44:14 on dist-test-slave-gkrl"
server_key: "75285e3024f51b8b7ab75933db8f0933"
server_key_iv: "681a7b98c47966446d75b0a11d9ab76f"
server_key_version: "encryptionkey@0"
I20260927 02:44:14.809139 3424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:14.811522 3439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:14.812770 3424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:14.812901 3424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "b0c7bad3b7834c27839505b991095080"
format_stamp: "Formatted at 2026-09-27 02:44:14 on dist-test-slave-gkrl"
server_key: "75285e3024f51b8b7ab75933db8f0933"
server_key_iv: "681a7b98c47966446d75b0a11d9ab76f"
server_key_version: "encryptionkey@0"
I20260927 02:44:14.813001 3424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:14.822305 3424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:14.825397 3424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:14.825594 3424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:14.832631 3424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42647
I20260927 02:44:14.832898 3491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42647 every 8 connection(s)
I20260927 02:44:14.833624 3424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:14.836146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 3424
I20260927 02:44:14.836251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:14.836310 3492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:14.836576 392 external_mini_cluster.cc:1428] Setting key 5f02741a0edf31a1509d7319f1a52319
I20260927 02:44:14.842218 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080: Bootstrap starting.
I20260927 02:44:14.844363 3492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:14.845085 3492 log.cc:824] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:44:14 dist-test-slave-gkrl krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:14.846967 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080: No bootstrap required, opened a new log
I20260927 02:44:14.851150 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:14.837900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.0.98.62:42647)
0927 02:44:14.838316 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:14.838331 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:14.838364 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:44:14.839256 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:14.839274 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:14.839597 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:14.839940 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:14.840775 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:14.841733 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:14.842689 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:14.843175 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:14.843361 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:14.846052 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:14.846057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:14.846070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:14.846094 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:14.848114 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:14.848525 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:14.848531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:14.848536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:14.848668 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:14.848920 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:14.848924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:14.848925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:14.849499 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:14.849709 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:14.849922 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 02:44:14.850215 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
I20260927 02:44:14.851594 3492 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c7bad3b7834c27839505b991095080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } }
I20260927 02:44:14.851794 3492 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:14.851859 3492 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c7bad3b7834c27839505b991095080, State: Initialized, Role: FOLLOWER
I20260927 02:44:14.852265 3492 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c7bad3b7834c27839505b991095080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } }
I20260927 02:44:14.852387 3492 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:14.852471 3492 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:14.852588 3492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:14.853511 3492 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c7bad3b7834c27839505b991095080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } }
I20260927 02:44:14.853838 3492 leader_election.cc:302] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c7bad3b7834c27839505b991095080; no voters:
I20260927 02:44:14.854116 3492 leader_election.cc:288] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:14.854228 3497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:14.854466 3497 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [term 1 LEADER]: Becoming Leader. State: Replica: b0c7bad3b7834c27839505b991095080, State: Running, Role: LEADER
I20260927 02:44:14.854789 3497 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c7bad3b7834c27839505b991095080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } }
I20260927 02:44:14.855173 3492 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:14.856364 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c7bad3b7834c27839505b991095080. Latest consensus state: current_term: 1 leader_uuid: "b0c7bad3b7834c27839505b991095080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c7bad3b7834c27839505b991095080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } } }
I20260927 02:44:14.856490 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:14.856695 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0c7bad3b7834c27839505b991095080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c7bad3b7834c27839505b991095080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } } }
I20260927 02:44:14.856827 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:14.856922 3506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:14.859668 3506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:14.864262 3506 catalog_manager.cc:1347] Generated new cluster ID: 702b3462b6d2462b8d48f3377fc81eef
I20260927 02:44:14.864318 3506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:14.878554 3506 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:14.879576 3506 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:14.885679 3506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0c7bad3b7834c27839505b991095080: Generated new TSK 0
I20260927 02:44:14.886361 3506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:14.915323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42647
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:15.022462 3518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:15.022691 3518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:15.022732 3518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:15.026698 3518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:15.026753 3518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:15.026839 3518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:15.031996 3518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:15.032961 3518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:15.033911 3518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:15.043054 3524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:15.043061 3523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:15.043051 3526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:15.043087 3518 server_base.cc:1034] running on GCE node
I20260927 02:44:15.043705 3518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:15.044348 3518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:15.045535 3518 hybrid_clock.cc:648] HybridClock initialized: now 1790477055045508 us; error 53 us; skew 500 ppm
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:15.048985 3518 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:15.050045 3518 webserver.cc:460] Webserver started at http://127.0.98.1:44535/ using document root <none> and password file <none>
I20260927 02:44:15.050557 3518 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:15.050644 3518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:15.050845 3518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:15.052771 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad6309b7709e4c0daf16605c751ef22a"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "22087b13c2b51fa769ebeed91f40b2f0"
server_key_iv: "a71079315c3821f750c3e71b903410bd"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.053246 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad6309b7709e4c0daf16605c751ef22a"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "22087b13c2b51fa769ebeed91f40b2f0"
server_key_iv: "a71079315c3821f750c3e71b903410bd"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.056694 3518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:15.058808 3533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.060034 3518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:15.060150 3518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "ad6309b7709e4c0daf16605c751ef22a"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "22087b13c2b51fa769ebeed91f40b2f0"
server_key_iv: "a71079315c3821f750c3e71b903410bd"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.060240 3518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:15.070744 3518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:15.073755 3518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:15.073946 3518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:15.074470 3518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:15.075248 3518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:15.075292 3518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.075353 3518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:15.075384 3518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.084282 3518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33489
I20260927 02:44:15.084303 3646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33489 every 8 connection(s)
I20260927 02:44:15.085129 3518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:15.091476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 3518
I20260927 02:44:15.091625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:15.091950 392 external_mini_cluster.cc:1428] Setting key 08225139e89f358d43c1c4f3356a98da
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:44:15.103132 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:15.086800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42969 (local address 127.0.98.62:42647)
0927 02:44:15.086959 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:15.086963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:15.087782 (+ 819us) server_negotiation.cc:407] Connection header received
0927 02:44:15.089223 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:15.089226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:15.089278 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:15.089395 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:15.090505 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.091041 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.092385 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.092703 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.092816 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:15.099534 (+ 6718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:15.099537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:15.099539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:15.099562 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:15.101070 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:15.101636 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:15.101639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:15.101641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:15.101724 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:15.102081 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:15.102084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:15.102086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:15.102265 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:15.102371 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:15.102846 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 02:44:15.102966 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:44:15.103853 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:15.087166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42647 (local address 127.0.98.1:42969)
0927 02:44:15.087631 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:15.087672 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:15.089019 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:15.089577 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:15.089583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:15.089918 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:15.090356 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:15.090369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.091164 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.091167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:15.092261 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:15.092270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.092846 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.092850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:15.093020 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:15.097930 (+ 4910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:15.097946 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:15.099355 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:15.101203 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:15.101209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:15.101223 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:15.101525 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:15.101824 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:15.101827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:15.101829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:15.101969 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:15.102369 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:15.102376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:15.102608 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 02:44:15.102792 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 02:44:15.103106 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":107,"threads_started":1}
I20260927 02:44:15.105253 3647 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42647
I20260927 02:44:15.105494 3647 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:15.105965 3647 heartbeater.cc:511] Master 127.0.98.62:42647 requested a full tablet report, sending...
I20260927 02:44:15.107676 3456 ts_manager.cc:194] Registered new tserver with Master: ad6309b7709e4c0daf16605c751ef22a (127.0.98.1:33489)
I20260927 02:44:15.109362 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:15.119015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42647
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:15.223256 3652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:15.223483 3652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:15.223523 3652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:15.227406 3652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:15.227455 3652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:15.227540 3652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:15.232503 3652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:15.233440 3652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:15.234714 3652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:15.245766 3657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:15.245771 3660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:15.245767 3658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:15.246135 3652 server_base.cc:1034] running on GCE node
I20260927 02:44:15.246611 3652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:15.247144 3652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:15.248303 3652 hybrid_clock.cc:648] HybridClock initialized: now 1790477055248282 us; error 32 us; skew 500 ppm
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:15.251004 3652 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:15.251726 3652 webserver.cc:460] Webserver started at http://127.0.98.2:46315/ using document root <none> and password file <none>
I20260927 02:44:15.252275 3652 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:15.252336 3652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:15.252493 3652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:15.254626 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b7f7d36de2844fead622b600c63ed67"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "a186d99d8882a502de70504b7d812004"
server_key_iv: "d25da541ab3f1c9046e7e58da98e8e20"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.255257 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b7f7d36de2844fead622b600c63ed67"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "a186d99d8882a502de70504b7d812004"
server_key_iv: "d25da541ab3f1c9046e7e58da98e8e20"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.259093 3652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 02:44:15.261660 3667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.262826 3652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:44:15.263088 3652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "9b7f7d36de2844fead622b600c63ed67"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "a186d99d8882a502de70504b7d812004"
server_key_iv: "d25da541ab3f1c9046e7e58da98e8e20"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.263217 3652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:15.282624 3652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:15.285971 3652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:15.286185 3652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:15.286761 3652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:15.287633 3652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:15.287684 3652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.287752 3652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:15.287791 3652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.297387 3652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43949
I20260927 02:44:15.297434 3780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43949 every 8 connection(s)
I20260927 02:44:15.298251 3652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:15.304984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 3652
I20260927 02:44:15.305145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:15.305421 392 external_mini_cluster.cc:1428] Setting key 8bacf3b7a2a88f28f45a7a6157ab0a2e
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:15.312523 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:15.299972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50541 (local address 127.0.98.62:42647)
0927 02:44:15.300112 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:15.300116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:15.301094 (+ 978us) server_negotiation.cc:407] Connection header received
0927 02:44:15.302751 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:15.302754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:15.302812 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:15.302924 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:15.304122 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.304644 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.305982 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.306255 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.306342 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:15.308919 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:15.308923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:15.308925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:15.308956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:15.310472 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:15.310973 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:15.310976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:15.310978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:15.311057 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:15.311396 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:15.311399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:15.311401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:15.311583 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:15.311687 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:15.312231 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 02:44:15.312359 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:44:15.313268 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:15.300456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42647 (local address 127.0.98.2:50541)
0927 02:44:15.300940 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:15.300982 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:15.302519 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:15.303117 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:15.303124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:15.303453 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:15.303977 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:15.303993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.304784 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.304787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:15.305832 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:15.305844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.306405 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.306408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:15.306615 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:15.307268 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:15.307284 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:15.308757 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:15.310598 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:15.310604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:15.310617 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:15.310866 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:15.311155 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:15.311158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:15.311160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:15.311295 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:15.311686 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:15.311692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:15.311992 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 02:44:15.312175 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 02:44:15.312485 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:15.314525 3781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42647
I20260927 02:44:15.314730 3781 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:15.315234 3781 heartbeater.cc:511] Master 127.0.98.62:42647 requested a full tablet report, sending...
I20260927 02:44:15.316280 3456 ts_manager.cc:194] Registered new tserver with Master: 9b7f7d36de2844fead622b600c63ed67 (127.0.98.2:43949)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:44:15.316985 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:15.331193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42647
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:15.441191 3786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:15.441417 3786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:15.441457 3786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:15.445362 3786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:15.445411 3786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:15.445504 3786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:15.450598 3786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:15.451555 3786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:15.452584 3786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:15.462054 3792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:15.462040 3794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:15.462036 3791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:15.462581 3786 server_base.cc:1034] running on GCE node
I20260927 02:44:15.463001 3786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:15.463732 3786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:15.464916 3786 hybrid_clock.cc:648] HybridClock initialized: now 1790477055464893 us; error 37 us; skew 500 ppm
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:15.467782 3786 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:15.468649 3786 webserver.cc:460] Webserver started at http://127.0.98.3:33955/ using document root <none> and password file <none>
I20260927 02:44:15.469214 3786 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:15.469300 3786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:15.469521 3786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:15.471414 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "94034596db87422180bbf635f8f50a8d"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "feef19bea3b5d232739a601faa8c24b0"
server_key_iv: "d381806403ed1e7d1412a961d43f458d"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.472035 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94034596db87422180bbf635f8f50a8d"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "feef19bea3b5d232739a601faa8c24b0"
server_key_iv: "d381806403ed1e7d1412a961d43f458d"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.475436 3786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:15.477823 3801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.478977 3786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:44:15.479107 3786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "94034596db87422180bbf635f8f50a8d"
format_stamp: "Formatted at 2026-09-27 02:44:15 on dist-test-slave-gkrl"
server_key: "feef19bea3b5d232739a601faa8c24b0"
server_key_iv: "d381806403ed1e7d1412a961d43f458d"
server_key_version: "encryptionkey@0"
I20260927 02:44:15.479209 3786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:15.488279 3786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:15.490819 3786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:15.491008 3786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:15.491524 3786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:15.492326 3786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:15.492398 3786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.492465 3786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:15.492515 3786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:15.501190 3786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38315
I20260927 02:44:15.501215 3914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38315 every 8 connection(s)
I20260927 02:44:15.501998 3786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:15.506726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 3786
I20260927 02:44:15.506850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:15.507121 392 external_mini_cluster.cc:1428] Setting key d4c53394899ff81859b04a3580a60e9a
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:15.514645 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:15.503657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36997 (local address 127.0.98.62:42647)
0927 02:44:15.503796 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:15.503800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:15.504628 (+ 828us) server_negotiation.cc:407] Connection header received
0927 02:44:15.505957 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:15.505960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:15.506014 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:15.506124 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:15.507346 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.507957 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.509059 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.509343 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.509425 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:15.511685 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:15.511687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:15.511689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:15.511712 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:15.512926 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:15.513360 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:15.513363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:15.513364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:15.513437 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:15.513724 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:15.513726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:15.513728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:15.513882 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:15.513956 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:15.514434 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 02:44:15.514535 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:15.515372 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:15.504048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42647 (local address 127.0.98.3:36997)
0927 02:44:15.504474 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:15.504513 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:15.505752 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:15.506301 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:15.506308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:15.506653 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:15.507179 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:15.507192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.508074 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.508079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:15.508920 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:15.508960 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.509473 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.509476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:15.509604 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:15.510168 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:15.510183 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:15.511545 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:15.513037 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:15.513041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:15.513051 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:15.513273 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:15.513529 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:15.513532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:15.513533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:15.513647 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:15.514028 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:15.514033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:15.514235 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 02:44:15.514390 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 02:44:15.514624 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260927 02:44:15.516646 3915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42647
I20260927 02:44:15.516919 3915 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:15.517473 3915 heartbeater.cc:511] Master 127.0.98.62:42647 requested a full tablet report, sending...
I20260927 02:44:15.518538 3456 ts_manager.cc:194] Registered new tserver with Master: 94034596db87422180bbf635f8f50a8d (127.0.98.3:38315)
I20260927 02:44:15.519079 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36997
I20260927 02:44:15.520797 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477055, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477055, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:15.545241 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:15.537517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.0.98.62:42647)
0927 02:44:15.537638 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:15.537642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:15.537669 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:44:15.537788 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:15.537791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:15.537828 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:15.537900 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:15.538577 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.539093 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.539926 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:15.540144 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:15.540225 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:15.542385 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:15.542388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:15.542390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:15.542411 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:15.543832 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:15.544271 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:15.544274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:15.544275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:15.544345 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:15.544570 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:15.544572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:15.544573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:15.544757 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:15.544853 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:15.545047 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 02:44:15.545131 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 02:44:15.545644 3456 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50694
I20260927 02:44:15.546446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 3518
I20260927 02:44:15.551738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 3652
I20260927 02:44:15.557169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 3786
I20260927 02:44:15.562150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 3424
2026-09-27T02:44:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2046 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3927](info): set up 2 sockets
Sep 27 02:44:15 dist-test-slave-gkrl krb5kdc[3927](info): commencing operation
krb5kdc: starting...
W20260927 02:44:16.604132 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:16 dist-test-slave-gkrl krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:16.720798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:16.826964 3940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:16.827248 3940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:16.827325 3940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:16.831900 3940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:16.831976 3940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:16.832013 3940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:16.832050 3940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:16.832093 3940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:16.837275 3940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:16.838352 3940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:16.839355 3940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:16.847671 3945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:16.847713 3948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:16.847697 3946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:16.847852 3940 server_base.cc:1034] running on GCE node
I20260927 02:44:16.848555 3940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:16.849556 3940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:16.850723 3940 hybrid_clock.cc:648] HybridClock initialized: now 1790477056850696 us; error 38 us; skew 500 ppm
Sep 27 02:44:16 dist-test-slave-gkrl krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:16.853345 3940 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:16.854099 3940 webserver.cc:460] Webserver started at http://127.0.98.62:35929/ using document root <none> and password file <none>
I20260927 02:44:16.854584 3940 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:16.854640 3940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:16.854791 3940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:16.856840 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "191dbb4cfc6b48fba46d6df1a2520702"
format_stamp: "Formatted at 2026-09-27 02:44:16 on dist-test-slave-gkrl"
server_key: "d711edf596da31de4326c55abf48f0b0"
server_key_iv: "9f0ff1f2981dab5d6847c1b4a5d07d63"
server_key_version: "encryptionkey@0"
I20260927 02:44:16.857318 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "191dbb4cfc6b48fba46d6df1a2520702"
format_stamp: "Formatted at 2026-09-27 02:44:16 on dist-test-slave-gkrl"
server_key: "d711edf596da31de4326c55abf48f0b0"
server_key_iv: "9f0ff1f2981dab5d6847c1b4a5d07d63"
server_key_version: "encryptionkey@0"
I20260927 02:44:16.860773 3940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:16.862967 3955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:16.864374 3940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:16.864491 3940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "191dbb4cfc6b48fba46d6df1a2520702"
format_stamp: "Formatted at 2026-09-27 02:44:16 on dist-test-slave-gkrl"
server_key: "d711edf596da31de4326c55abf48f0b0"
server_key_iv: "9f0ff1f2981dab5d6847c1b4a5d07d63"
server_key_version: "encryptionkey@0"
I20260927 02:44:16.864573 3940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:16.886982 3940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:16.889802 3940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:16.889946 3940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:16.896714 3940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44769
I20260927 02:44:16.896742 4007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44769 every 8 connection(s)
I20260927 02:44:16.897636 3940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:16.900457 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:16.905699 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702: Bootstrap starting.
I20260927 02:44:16.906637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 3940
I20260927 02:44:16.906773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:16.907073 392 external_mini_cluster.cc:1428] Setting key fd3bc7dfbcf01bf4690cef709562da9a
I20260927 02:44:16.908558 4008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:16.909258 4008 log.cc:824] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:16.911340 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702: No bootstrap required, opened a new log
I20260927 02:44:16.915719 4008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "191dbb4cfc6b48fba46d6df1a2520702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } }
I20260927 02:44:16.915974 4008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:16.916049 4008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191dbb4cfc6b48fba46d6df1a2520702, State: Initialized, Role: FOLLOWER
Sep 27 02:44:16 dist-test-slave-gkrl krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:16.916488 4008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "191dbb4cfc6b48fba46d6df1a2520702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } }
I20260927 02:44:16.916638 4008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:16.916702 4008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:16.916792 4008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:16.917871 4008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "191dbb4cfc6b48fba46d6df1a2520702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } }
I20260927 02:44:16.918203 4008 leader_election.cc:302] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 191dbb4cfc6b48fba46d6df1a2520702; no voters:
I20260927 02:44:16.918555 4008 leader_election.cc:288] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:16.918619 4013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:16.918807 4013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [term 1 LEADER]: Becoming Leader. State: Replica: 191dbb4cfc6b48fba46d6df1a2520702, State: Running, Role: LEADER
I20260927 02:44:16.919137 4013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "191dbb4cfc6b48fba46d6df1a2520702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } }
I20260927 02:44:16.919760 4008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:16.920797 4015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 191dbb4cfc6b48fba46d6df1a2520702. Latest consensus state: current_term: 1 leader_uuid: "191dbb4cfc6b48fba46d6df1a2520702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "191dbb4cfc6b48fba46d6df1a2520702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } } }
I20260927 02:44:16.920910 4015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:16.921293 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "191dbb4cfc6b48fba46d6df1a2520702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "191dbb4cfc6b48fba46d6df1a2520702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } } }
I20260927 02:44:16.922101 4022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:16.922185 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:16.922533 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:16.908182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.0.98.62:44769)
0927 02:44:16.908569 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:16.908584 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:16.908640 (+ 56us) server_negotiation.cc:407] Connection header received
0927 02:44:16.909277 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:16.909295 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:16.909705 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:16.910052 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:16.910923 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:16.912038 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:16.913258 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:16.913716 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:16.913836 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:16.916914 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:16.916918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:16.916930 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:16.916954 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:16.918772 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:16.919267 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:16.919273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:16.919278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:16.919403 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:16.919712 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:16.919715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:16.919717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:16.920270 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:16.921098 (+ 828us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:16.921236 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:44:16.921540 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260927 02:44:16.925330 4022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:16.930730 4022 catalog_manager.cc:1347] Generated new cluster ID: 88875dd6950f4a3fb50db8edaef34dff
I20260927 02:44:16.930804 4022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:16.957891 4022 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:16.958868 4022 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:16.966933 4022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 191dbb4cfc6b48fba46d6df1a2520702: Generated new TSK 0
I20260927 02:44:16.967710 4022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:17.005462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:17.145597 4034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:17.145834 4034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:17.145874 4034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:17.149796 4034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:17.149845 4034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:17.149899 4034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:17.154906 4034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:17.155792 4034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:17.156862 4034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:17.166220 4042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:17.166227 4039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:17.166247 4040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:17.166589 4034 server_base.cc:1034] running on GCE node
I20260927 02:44:17.167099 4034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:17.167801 4034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:17.168982 4034 hybrid_clock.cc:648] HybridClock initialized: now 1790477057168970 us; error 32 us; skew 500 ppm
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:17.171710 4034 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:17.172590 4034 webserver.cc:460] Webserver started at http://127.0.98.1:46045/ using document root <none> and password file <none>
I20260927 02:44:17.173096 4034 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:17.173154 4034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:17.173311 4034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:17.175216 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "fd0f1b44f59e46aeb2110aa31f82a909"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "d2f4bf276422ba0efd47b13abe0306b8"
server_key_iv: "5b69d37deaa48a767488b73325f8cf60"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.175716 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fd0f1b44f59e46aeb2110aa31f82a909"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "d2f4bf276422ba0efd47b13abe0306b8"
server_key_iv: "5b69d37deaa48a767488b73325f8cf60"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.179286 4034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:17.181670 4049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.182874 4034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:44:17.182971 4034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "fd0f1b44f59e46aeb2110aa31f82a909"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "d2f4bf276422ba0efd47b13abe0306b8"
server_key_iv: "5b69d37deaa48a767488b73325f8cf60"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.183080 4034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:17.210279 4034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:17.213577 4034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:17.213733 4034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:17.214278 4034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:17.215070 4034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:17.215116 4034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.215155 4034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:17.215170 4034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.223927 4034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34283
I20260927 02:44:17.223963 4162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34283 every 8 connection(s)
I20260927 02:44:17.224857 4034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:17.232661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 4034
I20260927 02:44:17.232800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:17.233240 392 external_mini_cluster.cc:1428] Setting key f8de950d4e089024d76d9b1094292c92
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:17.239238 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:17.226662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41535 (local address 127.0.98.62:44769)
0927 02:44:17.226800 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:17.226809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:17.227537 (+ 728us) server_negotiation.cc:407] Connection header received
0927 02:44:17.228961 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:17.228965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:17.229046 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:17.229142 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:17.230252 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.230773 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.231931 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.232188 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.232303 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:17.235073 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:17.235076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:17.235078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:17.235102 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:17.236698 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.237332 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.237335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.237336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.237451 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.237908 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.237915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.237920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.238194 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:17.238315 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:17.238840 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 02:44:17.239017 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:17.240032 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:17.226968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44769 (local address 127.0.98.1:41535)
0927 02:44:17.227384 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:17.227424 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:17.228746 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:17.229310 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:17.229316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:17.229662 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:17.230113 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:17.230125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.230891 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.230895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:17.231777 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:17.231785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.232336 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.232340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:17.232531 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:17.233110 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:17.233132 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:17.234876 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:17.236821 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:17.236828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:17.236845 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:17.237219 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:17.237611 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:17.237614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:17.237615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:17.237754 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:17.238331 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:17.238337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:17.238565 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 02:44:17.238771 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 02:44:17.239074 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260927 02:44:17.241357 4163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44769
I20260927 02:44:17.241641 4163 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:17.242265 4163 heartbeater.cc:511] Master 127.0.98.62:44769 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:44:17.243960 3972 ts_manager.cc:194] Registered new tserver with Master: fd0f1b44f59e46aeb2110aa31f82a909 (127.0.98.1:34283)
I20260927 02:44:17.245128 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:17.263450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:17.371179 4168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:17.371439 4168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:17.371526 4168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:17.375488 4168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:17.375574 4168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:17.375669 4168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:17.380904 4168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:17.381851 4168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:17.382787 4168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:17.391865 4173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:17.391969 4176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:17.392194 4174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:17.392318 4168 server_base.cc:1034] running on GCE node
I20260927 02:44:17.392684 4168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:17.393260 4168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:17.394446 4168 hybrid_clock.cc:648] HybridClock initialized: now 1790477057394429 us; error 28 us; skew 500 ppm
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:17.397099 4168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:17.397806 4168 webserver.cc:460] Webserver started at http://127.0.98.2:44511/ using document root <none> and password file <none>
I20260927 02:44:17.398281 4168 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:17.398339 4168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:17.398550 4168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:17.400605 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "1dcd2151fd5f4e37bc8ca24afc430cc4"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "41c7bc941d8f5548f049ec7b8e312050"
server_key_iv: "19d46cbd3a61ce25cd65e7087db0e8a9"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.401218 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1dcd2151fd5f4e37bc8ca24afc430cc4"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "41c7bc941d8f5548f049ec7b8e312050"
server_key_iv: "19d46cbd3a61ce25cd65e7087db0e8a9"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.404630 4168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:44:17.406737 4183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.407945 4168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:17.408038 4168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "1dcd2151fd5f4e37bc8ca24afc430cc4"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "41c7bc941d8f5548f049ec7b8e312050"
server_key_iv: "19d46cbd3a61ce25cd65e7087db0e8a9"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.408155 4168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:17.422039 4168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:17.424687 4168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:17.424866 4168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:17.425387 4168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:17.426128 4168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:17.426172 4168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.426236 4168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:17.426273 4168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.434633 4168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34531
I20260927 02:44:17.434657 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34531 every 8 connection(s)
I20260927 02:44:17.435444 4168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:17.439034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 4168
I20260927 02:44:17.439159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:17.439404 392 external_mini_cluster.cc:1428] Setting key 6bed96be37a57f62da63c651a41b0a7a
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:17.450860 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:17.438748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33239 (local address 127.0.98.62:44769)
0927 02:44:17.438908 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:17.438913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:17.438931 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:44:17.440009 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:17.440040 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:17.440105 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:17.440190 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:17.441356 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.442192 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.443980 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.444203 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.444291 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:17.447114 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:17.447118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:17.447120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:17.447154 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:17.448777 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.449299 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.449306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.449309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.449390 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.449742 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.449745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.449747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.450004 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:17.450110 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:17.450585 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 02:44:17.450696 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:44:17.451725 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:17.437842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44769 (local address 127.0.98.2:33239)
0927 02:44:17.438307 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:17.438346 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:17.439750 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:17.440319 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:17.440325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:17.440750 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:17.441213 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:17.441225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.442348 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.442354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:17.443836 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:17.443850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.444358 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.444364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:17.444607 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:17.445269 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:17.445291 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:17.446953 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:17.448912 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:17.448917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:17.448933 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:17.449195 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:17.449487 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:17.449491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:17.449493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:17.449632 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:17.450118 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:17.450126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:17.450351 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 02:44:17.450545 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 02:44:17.450833 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":152,"threads_started":1}
I20260927 02:44:17.452975 4297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44769
I20260927 02:44:17.453218 4297 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:17.453660 4297 heartbeater.cc:511] Master 127.0.98.62:44769 requested a full tablet report, sending...
I20260927 02:44:17.454645 3972 ts_manager.cc:194] Registered new tserver with Master: 1dcd2151fd5f4e37bc8ca24afc430cc4 (127.0.98.2:34531)
I20260927 02:44:17.455269 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:17.466913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:17.572703 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:17.572974 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:17.573091 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:17.576988 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:17.577069 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:17.577157 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:17.582136 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:17.583098 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:17.584117 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:17.593354 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:17.593338 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:17.593446 4302 server_base.cc:1034] running on GCE node
W20260927 02:44:17.593339 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:17.593963 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:17.594616 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:17.595793 4302 hybrid_clock.cc:648] HybridClock initialized: now 1790477057595782 us; error 29 us; skew 500 ppm
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:17.598382 4302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:17.599119 4302 webserver.cc:460] Webserver started at http://127.0.98.3:38877/ using document root <none> and password file <none>
I20260927 02:44:17.599619 4302 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:17.599679 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:17.599834 4302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:17.602065 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "53759f48ee1642b9a065ed8c893fb8f7"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "7c9f07220ae13bcdf4f77645da4c53f1"
server_key_iv: "b2ca9f099137dbd2481fab8196ab6c5d"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.602552 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "53759f48ee1642b9a065ed8c893fb8f7"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "7c9f07220ae13bcdf4f77645da4c53f1"
server_key_iv: "b2ca9f099137dbd2481fab8196ab6c5d"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.605983 4302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:17.608335 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.609546 4302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:17.609637 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "53759f48ee1642b9a065ed8c893fb8f7"
format_stamp: "Formatted at 2026-09-27 02:44:17 on dist-test-slave-gkrl"
server_key: "7c9f07220ae13bcdf4f77645da4c53f1"
server_key_iv: "b2ca9f099137dbd2481fab8196ab6c5d"
server_key_version: "encryptionkey@0"
I20260927 02:44:17.609719 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:17.631304 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:17.634476 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:17.634636 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:17.635135 4302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:17.635860 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:17.635926 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.635962 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:17.635983 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:17.644454 4302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43127
I20260927 02:44:17.644526 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43127 every 8 connection(s)
I20260927 02:44:17.645391 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:17.652633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 4302
I20260927 02:44:17.652760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:17.653069 392 external_mini_cluster.cc:1428] Setting key 56b52d0820cb11e7dedd5c6ff06679db
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:17.658802 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:17.647158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54251 (local address 127.0.98.62:44769)
0927 02:44:17.647302 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:17.647306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:17.648131 (+ 825us) server_negotiation.cc:407] Connection header received
0927 02:44:17.649504 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:17.649507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:17.649565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:17.649674 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:17.650779 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.651304 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.652501 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.652776 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.652880 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:17.655769 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:17.655772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:17.655774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:17.655797 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:17.657059 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.657508 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.657511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.657512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.657584 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.657857 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.657859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.657861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.658020 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:17.658093 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:17.658589 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 02:44:17.658690 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:44:17.659538 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:17.647481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44769 (local address 127.0.98.3:54251)
0927 02:44:17.647979 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:17.648020 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:17.649287 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:17.649837 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:17.649844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:17.650185 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:17.650634 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:17.650646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.651438 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.651447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:17.652377 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:17.652385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.652947 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.652951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:17.653162 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:17.653799 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:17.653817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:17.655619 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:17.657168 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:17.657172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:17.657182 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:17.657415 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:17.657661 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:17.657663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:17.657664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:17.657782 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:17.658102 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:17.658107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:17.658310 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 02:44:17.658488 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 02:44:17.658726 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
I20260927 02:44:17.660704 4431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44769
I20260927 02:44:17.660897 4431 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:17.661303 4431 heartbeater.cc:511] Master 127.0.98.62:44769 requested a full tablet report, sending...
I20260927 02:44:17.662204 3972 ts_manager.cc:194] Registered new tserver with Master: 53759f48ee1642b9a065ed8c893fb8f7 (127.0.98.3:43127)
I20260927 02:44:17.662724 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54251
I20260927 02:44:17.667021 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:44:17.692456 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:17.682865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.0.98.1:34283)
0927 02:44:17.683127 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:17.683142 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:17.683169 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:44:17.683291 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:17.683297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:17.683477 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:17.683635 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:17.684376 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.685151 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.685970 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.686425 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.686547 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:17.689019 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:17.689028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:17.689045 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:17.689073 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:17.690706 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.691145 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.691151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.691152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.691228 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.691504 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.691507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.691509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.691732 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:17.691966 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:17.692142 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:44:17.692288 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":60,"threads_started":1}
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:17.702888 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:17.695468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.0.98.62:44769)
0927 02:44:17.695597 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:17.695601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:17.695616 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:44:17.695730 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:17.695732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:17.695769 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:17.695833 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:17.696474 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.696985 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.697694 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:17.697893 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:17.697965 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:17.700193 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:17.700199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:17.700202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:17.700228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:17.701571 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.701926 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.701928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.701929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.701997 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:17.702240 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:17.702242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:17.702243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:17.702426 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:17.702537 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:17.702697 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 02:44:17.702778 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 02:44:17.703253 3972 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43564
I20260927 02:44:17.703958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 4034
I20260927 02:44:17.710556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 4168
I20260927 02:44:17.715816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 4302
I20260927 02:44:17.720978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 3940
2026-09-27T02:44:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2158 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[4443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[4443](info): set up 2 sockets
Sep 27 02:44:17 dist-test-slave-gkrl krb5kdc[4443](info): commencing operation
krb5kdc: starting...
W20260927 02:44:18.756047 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:18 dist-test-slave-gkrl krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:18.874951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:18.978734 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:18.979113 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:18.979200 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:18.983155 4456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:18.983245 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:18.983285 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:18.983322 4456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:18.983354 4456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:18.988432 4456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:18.989429 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:18.990406 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:44:18.998672 4456 server_base.cc:1034] running on GCE node
W20260927 02:44:18.998627 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:18.998626 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:18.998631 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:18.999545 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:19.000772 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:19.002007 4456 hybrid_clock.cc:648] HybridClock initialized: now 1790477059001975 us; error 46 us; skew 500 ppm
Sep 27 02:44:19 dist-test-slave-gkrl krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:19.005039 4456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:19.005684 4456 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:19.005746 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:19.005898 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:19.007827 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "7a39c4a8f73d412baabc3baadbdcbdc7"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "d94ce30f7e2970d79f5cf04ec543ff1d"
server_key_iv: "47bbd8adf44241453b63a2f70ec110ec"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.008394 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a39c4a8f73d412baabc3baadbdcbdc7"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "d94ce30f7e2970d79f5cf04ec543ff1d"
server_key_iv: "47bbd8adf44241453b63a2f70ec110ec"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.011770 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:19.014161 4470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.015398 4456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:19.015494 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "7a39c4a8f73d412baabc3baadbdcbdc7"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "d94ce30f7e2970d79f5cf04ec543ff1d"
server_key_iv: "47bbd8adf44241453b63a2f70ec110ec"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.015575 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:19.039108 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:19.042510 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:19.042670 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:19.049657 4456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37813
I20260927 02:44:19.049949 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37813 every 8 connection(s)
I20260927 02:44:19.050469 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:19.053197 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:19.058403 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7: Bootstrap starting.
I20260927 02:44:19.060529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 4456
I20260927 02:44:19.060654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:19.060958 392 external_mini_cluster.cc:1428] Setting key f366c92554035afdb576da64ef69d537
I20260927 02:44:19.060998 4523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:19.061841 4523 log.cc:824] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:19.064201 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7: No bootstrap required, opened a new log
I20260927 02:44:19.068277 4523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39c4a8f73d412baabc3baadbdcbdc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260927 02:44:19.068512 4523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:19.068584 4523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a39c4a8f73d412baabc3baadbdcbdc7, State: Initialized, Role: FOLLOWER
I20260927 02:44:19.069098 4523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39c4a8f73d412baabc3baadbdcbdc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260927 02:44:19.069254 4523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:19.069319 4523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:19.069410 4523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:19.070291 4523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39c4a8f73d412baabc3baadbdcbdc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260927 02:44:19.070623 4523 leader_election.cc:302] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a39c4a8f73d412baabc3baadbdcbdc7; no voters:
Sep 27 02:44:19 dist-test-slave-gkrl krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:19.071003 4523 leader_election.cc:288] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:19.071060 4528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:19.071244 4528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [term 1 LEADER]: Becoming Leader. State: Replica: 7a39c4a8f73d412baabc3baadbdcbdc7, State: Running, Role: LEADER
I20260927 02:44:19.071596 4528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39c4a8f73d412baabc3baadbdcbdc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260927 02:44:19.072268 4523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:19.073277 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a39c4a8f73d412baabc3baadbdcbdc7. Latest consensus state: current_term: 1 leader_uuid: "7a39c4a8f73d412baabc3baadbdcbdc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39c4a8f73d412baabc3baadbdcbdc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } } }
I20260927 02:44:19.073727 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:19.073614 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a39c4a8f73d412baabc3baadbdcbdc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39c4a8f73d412baabc3baadbdcbdc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } } }
I20260927 02:44:19.074100 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7 [sys.catalog]: This master's current role is: LEADER
W20260927 02:44:19.077426 4543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:44:19.077534 4543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:44:19.077631 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:19.078056 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.062231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.0.98.62:37813)
0927 02:44:19.062928 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.062947 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.063016 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:44:19.063928 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.063953 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.064377 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.064758 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:19.065595 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.066583 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.067947 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.068434 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.068634 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.071436 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:19.071442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:19.071458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:19.071489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:19.074860 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.075332 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.075338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.075344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.075463 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.075760 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.075763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.075765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.076313 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:19.076512 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.076806 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.077098 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":115,"threads_started":1}
I20260927 02:44:19.078413 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:19.083841 4540 catalog_manager.cc:1347] Generated new cluster ID: e53a1a988fe34c92b76adb24fe8f46b4
I20260927 02:44:19.083940 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:19.091648 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:19.092840 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:19.099823 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7: Generated new TSK 0
I20260927 02:44:19.100471 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:19.135226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:19.250308 4549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:19.250542 4549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:19.250586 4549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:19.254477 4549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:19.254527 4549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:19.254583 4549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:19.259414 4549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:19.260368 4549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:19.261269 4549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:19.270572 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:19.270638 4549 server_base.cc:1034] running on GCE node
W20260927 02:44:19.270586 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:19.270596 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:19.271354 4549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:19.272079 4549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:19.273289 4549 hybrid_clock.cc:648] HybridClock initialized: now 1790477059273260 us; error 43 us; skew 500 ppm
Sep 27 02:44:19 dist-test-slave-gkrl krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:19.276443 4549 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:19.277123 4549 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:19.277216 4549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:19.277448 4549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:19.279387 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f4306545e814232b6788a60740bef1e"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "265adfa6ab7ed275b5866d6823c4d840"
server_key_iv: "70913804c64085fa5267418bde706482"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.280002 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f4306545e814232b6788a60740bef1e"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "265adfa6ab7ed275b5866d6823c4d840"
server_key_iv: "70913804c64085fa5267418bde706482"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.283491 4549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:44:19.285809 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.286942 4549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:19.287076 4549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "2f4306545e814232b6788a60740bef1e"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "265adfa6ab7ed275b5866d6823c4d840"
server_key_iv: "70913804c64085fa5267418bde706482"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.287179 4549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:19.305568 4549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:19.308338 4549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:19.308530 4549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:19.309052 4549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:19.309805 4549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:19.309872 4549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.309940 4549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:19.309995 4549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.319082 4549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45065
I20260927 02:44:19.319098 4676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45065 every 8 connection(s)
I20260927 02:44:19.319674 4549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:19.321544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 4549
I20260927 02:44:19.321668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:19.321962 392 external_mini_cluster.cc:1428] Setting key 0c70f58c8154f85f9fac474209eef26a
Sep 27 02:44:19 dist-test-slave-gkrl krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:44:19.334954 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.321358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55699 (local address 127.0.98.62:37813)
0927 02:44:19.321515 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.321519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.322465 (+ 946us) server_negotiation.cc:407] Connection header received
0927 02:44:19.324034 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.324037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.324082 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.324159 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:19.325600 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.326135 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.327958 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.328185 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.328276 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.331102 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:19.331105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:19.331107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:19.331130 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:19.332736 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.333227 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.333230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.333231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.333303 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.333668 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.333673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.333674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.333865 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:19.333945 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.334635 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.334770 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:44:19.335750 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.321809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37813 (local address 127.0.98.1:55699)
0927 02:44:19.322301 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:19.322362 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:19.323750 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:19.324357 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:19.324369 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:19.324879 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:19.325459 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.325472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.326271 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.326277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.327800 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.327815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.328344 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.328351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.328590 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.329253 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:19.329281 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:19.330922 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:19.332870 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:19.332875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:19.332887 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:19.333120 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:19.333384 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:19.333386 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:19.333387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:19.333513 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:19.333999 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:19.334004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:19.334285 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 02:44:19.334550 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 02:44:19.334865 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
I20260927 02:44:19.337085 4677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37813
I20260927 02:44:19.337319 4677 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:19.337751 4677 heartbeater.cc:511] Master 127.0.98.62:37813 requested a full tablet report, sending...
I20260927 02:44:19.339248 4487 ts_manager.cc:194] Registered new tserver with Master: 2f4306545e814232b6788a60740bef1e (127.0.98.1:45065)
I20260927 02:44:19.340401 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:19.348937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:19.455410 4682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:19.455698 4682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:19.455770 4682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:19.459779 4682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:19.459872 4682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:19.460006 4682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:19.465075 4682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:19.466037 4682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:19.467015 4682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:19.476269 4687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:19.476312 4690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:19.476279 4688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:19.476589 4682 server_base.cc:1034] running on GCE node
I20260927 02:44:19.476948 4682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:19.477540 4682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:19.478717 4682 hybrid_clock.cc:648] HybridClock initialized: now 1790477059478715 us; error 50 us; skew 500 ppm
Sep 27 02:44:19 dist-test-slave-gkrl krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:19.481441 4682 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:19.482136 4682 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:19.482197 4682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:19.482344 4682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:19.484288 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "96c9d6640afd45cc81940fe17cfb29b2"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "b875965a9ab5eb262b3429767875c990"
server_key_iv: "bde6a0d0fd3b3c87d968993fce7428e6"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.484772 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96c9d6640afd45cc81940fe17cfb29b2"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "b875965a9ab5eb262b3429767875c990"
server_key_iv: "bde6a0d0fd3b3c87d968993fce7428e6"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.488174 4682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:19.490412 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.491676 4682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:19.491804 4682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "96c9d6640afd45cc81940fe17cfb29b2"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "b875965a9ab5eb262b3429767875c990"
server_key_iv: "bde6a0d0fd3b3c87d968993fce7428e6"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.491930 4682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:19.508471 4682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:19.511085 4682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:19.511268 4682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:19.511775 4682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:19.512557 4682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:19.512629 4682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.512694 4682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:19.512745 4682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.521526 4682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43789
I20260927 02:44:19.521550 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43789 every 8 connection(s)
I20260927 02:44:19.522068 4682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:19.524333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 4682
I20260927 02:44:19.524468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:19.524767 392 external_mini_cluster.cc:1428] Setting key 925fbc70b09fc10c011e035c525fe3ba
Sep 27 02:44:19 dist-test-slave-gkrl krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:19.535763 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.523668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55827 (local address 127.0.98.62:37813)
0927 02:44:19.523798 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.523802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.524702 (+ 900us) server_negotiation.cc:407] Connection header received
0927 02:44:19.526419 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.526422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.526474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.526549 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:19.527664 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.528264 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.529383 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.529610 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.529720 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.532180 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:19.532184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:19.532186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:19.532217 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:19.533741 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.534252 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.534257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.534259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.534347 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.534693 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.534698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.534700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.534872 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:19.535028 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.535486 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.535599 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:44:19.536567 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.524127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37813 (local address 127.0.98.2:55827)
0927 02:44:19.524564 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:19.524613 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:19.526187 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:19.526711 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:19.526717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:19.527070 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:19.527512 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.527524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.528399 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.528402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.529257 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.529266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.529749 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.529753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.529923 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.530526 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:19.530541 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:19.532025 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:19.533857 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:19.533862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:19.533876 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:19.534147 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:19.534439 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:19.534444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:19.534446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:19.534589 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:19.535026 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:19.535033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:19.535266 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 02:44:19.535443 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 02:44:19.535736 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":120,"threads_started":1}
I20260927 02:44:19.537685 4810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37813
I20260927 02:44:19.537918 4810 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:19.538393 4810 heartbeater.cc:511] Master 127.0.98.62:37813 requested a full tablet report, sending...
I20260927 02:44:19.539446 4487 ts_manager.cc:194] Registered new tserver with Master: 96c9d6640afd45cc81940fe17cfb29b2 (127.0.98.2:43789)
I20260927 02:44:19.540299 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:19.551527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:19.658720 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:19.658986 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:19.659060 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:19.663007 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:19.663092 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:19.663180 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:19.668252 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:19.669178 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:19.670113 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:19.683308 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:19.683336 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:19.683333 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:19.683260 4815 server_base.cc:1034] running on GCE node
I20260927 02:44:19.684008 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:19.684554 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:19.685716 4815 hybrid_clock.cc:648] HybridClock initialized: now 1790477059685697 us; error 30 us; skew 500 ppm
Sep 27 02:44:19 dist-test-slave-gkrl krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:19.688449 4815 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:19.689134 4815 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:19.689193 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:19.689343 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:19.691223 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb69fc1360134804a11b4fbe18c6a3fc"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "5226d57eccc10c165f0a2dd52c67c859"
server_key_iv: "9a36ba51006d48f59d73db3c6b434aad"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.691759 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb69fc1360134804a11b4fbe18c6a3fc"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "5226d57eccc10c165f0a2dd52c67c859"
server_key_iv: "9a36ba51006d48f59d73db3c6b434aad"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.695241 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 02:44:19.697749 4829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.698889 4815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:44:19.698989 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "bb69fc1360134804a11b4fbe18c6a3fc"
format_stamp: "Formatted at 2026-09-27 02:44:19 on dist-test-slave-gkrl"
server_key: "5226d57eccc10c165f0a2dd52c67c859"
server_key_iv: "9a36ba51006d48f59d73db3c6b434aad"
server_key_version: "encryptionkey@0"
I20260927 02:44:19.699064 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:19.711374 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:19.714277 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:19.714411 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:19.714879 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:19.715590 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:19.715636 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.715672 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:19.715693 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:19.724391 4815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40253
I20260927 02:44:19.724409 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40253 every 8 connection(s)
I20260927 02:44:19.724920 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:19.727142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 4815
I20260927 02:44:19.727250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:19.727523 392 external_mini_cluster.cc:1428] Setting key 780cff54e6eb263c752007ff064de273
Sep 27 02:44:19 dist-test-slave-gkrl krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:19.738271 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.726510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34603 (local address 127.0.98.62:37813)
0927 02:44:19.726668 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.726671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.727591 (+ 920us) server_negotiation.cc:407] Connection header received
0927 02:44:19.729216 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.729219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.729273 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.729384 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:19.730485 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.731014 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.732243 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.732466 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.732577 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.734851 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:19.734859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:19.734861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:19.734883 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:19.736347 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.736855 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.736860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.736862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.736942 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.737270 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.737275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.737277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.737452 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:19.737558 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.737999 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.738109 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 02:44:19.738978 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.726865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37813 (local address 127.0.98.3:34603)
0927 02:44:19.727436 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:19.727476 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:19.729020 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:19.729554 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:19.729560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:19.729908 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:19.730343 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.730356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.731136 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.731141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.732123 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.732133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.732607 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.732611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.732779 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.733361 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:19.733377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:19.734687 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:19.736484 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:19.736490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:19.736504 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:19.736749 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:19.737040 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:19.737043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:19.737045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:19.737176 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:19.737555 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:19.737560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:19.737783 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 02:44:19.737959 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 02:44:19.738241 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":129,"threads_started":1}
I20260927 02:44:19.739976 4943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37813
I20260927 02:44:19.740199 4943 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:19.740638 4943 heartbeater.cc:511] Master 127.0.98.62:37813 requested a full tablet report, sending...
I20260927 02:44:19.741569 4487 ts_manager.cc:194] Registered new tserver with Master: bb69fc1360134804a11b4fbe18c6a3fc (127.0.98.3:40253)
I20260927 02:44:19.742079 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34603
I20260927 02:44:19.752650 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:19.761873 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.754534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.0.98.62:37813)
0927 02:44:19.754680 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.754684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.754828 (+ 144us) server_negotiation.cc:407] Connection header received
0927 02:44:19.754996 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.754999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.755044 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.755126 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:19.755782 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.756357 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.757170 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.757378 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.757484 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.758786 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:19.758790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:19.758793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:19.758815 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:19.760280 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.760692 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.760696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.760698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.760774 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:19.761045 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:19.761050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:19.761051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:19.761246 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:19.761382 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.761609 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.761721 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:44:19.764539 4487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:19.767392 4487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:19.781258 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.775394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40253 (local address 127.0.0.1:57402)
0927 02:44:19.775715 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:19.775772 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:19.775917 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:19.776886 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:19.776894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:19.776918 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:19.777075 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.777083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.778449 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.778452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.779682 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.779692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.780738 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.780741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.780853 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.780886 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 02:44:19.781007 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 02:44:19.781119 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260927 02:44:19.781265 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.775527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.0.98.3:40253)
0927 02:44:19.776362 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.776377 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.776404 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:44:19.776486 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.776494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.776710 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.777233 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:19.777381 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.778317 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.779790 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.780618 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.780741 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.780933 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.781033 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.781119 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":188,"threads_started":1}
I20260927 02:44:19.781424 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.776116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43789 (local address 127.0.0.1:54232)
0927 02:44:19.776644 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:19.776657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:19.776745 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:19.777447 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:19.777450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:19.777466 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:19.777566 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.777573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.778903 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.778906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.780035 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.780042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.781018 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.781022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.781121 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.781146 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:44:19.781219 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:44:19.781284 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":91,"threads_started":1}
I20260927 02:44:19.781759 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.776293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.0.98.2:43789)
0927 02:44:19.777050 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.777065 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.777089 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:44:19.777164 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.777169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.777314 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.777467 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:19.777666 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.778762 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.780146 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.780734 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.780838 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.780959 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.781516 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.781609 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":90,"threads_started":1}
I20260927 02:44:19.782121 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.776117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45065 (local address 127.0.0.1:55480)
0927 02:44:19.776959 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:19.776971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:19.777068 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:19.777684 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:19.777687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:19.777704 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:19.777784 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.777789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.779234 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.779240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.780465 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.780476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.781673 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.781679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.781776 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.781808 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:44:19.781885 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:44:19.781967 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":200,"thread_start_us":90,"threads_started":1}
I20260927 02:44:19.782594 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.776364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.0.98.1:45065)
0927 02:44:19.776835 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.776847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.777077 (+ 230us) server_negotiation.cc:407] Connection header received
0927 02:44:19.777181 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.777186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.777374 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.777523 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:19.777919 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.779075 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.780605 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.781372 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.781476 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.781613 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.782330 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.782423 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":102,"threads_started":1}
I20260927 02:44:19.784415 4877 tablet_service.cc:1467] Processing CreateTablet for tablet 74638802181849c090e11c47f229fae7 (DEFAULT_TABLE table=test-table [id=9d2c4cd27dad40e5ba12c0cce10e9ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:19.784421 4744 tablet_service.cc:1467] Processing CreateTablet for tablet 74638802181849c090e11c47f229fae7 (DEFAULT_TABLE table=test-table [id=9d2c4cd27dad40e5ba12c0cce10e9ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:19.785315 4877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74638802181849c090e11c47f229fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:19.785305 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74638802181849c090e11c47f229fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:19.789039 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 74638802181849c090e11c47f229fae7 (DEFAULT_TABLE table=test-table [id=9d2c4cd27dad40e5ba12c0cce10e9ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:19.790014 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74638802181849c090e11c47f229fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:19.792496 4961 tablet_bootstrap.cc:492] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Bootstrap starting.
I20260927 02:44:19.792544 4960 tablet_bootstrap.cc:492] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Bootstrap starting.
I20260927 02:44:19.794255 4960 tablet_bootstrap.cc:654] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:19.794517 4961 tablet_bootstrap.cc:654] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:19.795469 4960 log.cc:824] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:19.795842 4961 log.cc:824] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:19.797864 4961 tablet_bootstrap.cc:492] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: No bootstrap required, opened a new log
I20260927 02:44:19.798091 4961 ts_tablet_manager.cc:1397] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:44:19.798720 4960 tablet_bootstrap.cc:492] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: No bootstrap required, opened a new log
I20260927 02:44:19.798897 4960 ts_tablet_manager.cc:1397] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:44:19.800007 4964 tablet_bootstrap.cc:492] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e: Bootstrap starting.
I20260927 02:44:19.802012 4964 tablet_bootstrap.cc:654] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:19.802978 4964 log.cc:824] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:19.804894 4964 tablet_bootstrap.cc:492] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e: No bootstrap required, opened a new log
I20260927 02:44:19.805096 4964 ts_tablet_manager.cc:1397] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:44:19.806049 4960 raft_consensus.cc:348] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:19.806305 4960 raft_consensus.cc:374] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:19.806342 4960 raft_consensus.cc:729] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb69fc1360134804a11b4fbe18c6a3fc, State: Initialized, Role: FOLLOWER
I20260927 02:44:19.806957 4960 consensus_queue.cc:260] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:19.808320 4960 ts_tablet_manager.cc:1428] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260927 02:44:19.808790 4943 heartbeater.cc:503] Master 127.0.98.62:37813 was elected leader, sending a full tablet report...
I20260927 02:44:19.809125 4964 raft_consensus.cc:348] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:19.809341 4964 raft_consensus.cc:374] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:19.809399 4964 raft_consensus.cc:729] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4306545e814232b6788a60740bef1e, State: Initialized, Role: FOLLOWER
I20260927 02:44:19.809815 4964 consensus_queue.cc:260] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:19.810297 4961 raft_consensus.cc:348] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:19.810549 4961 raft_consensus.cc:374] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:19.810609 4961 raft_consensus.cc:729] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96c9d6640afd45cc81940fe17cfb29b2, State: Initialized, Role: FOLLOWER
I20260927 02:44:19.810971 4964 ts_tablet_manager.cc:1428] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:44:19.811018 4961 consensus_queue.cc:260] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:19.811372 4677 heartbeater.cc:503] Master 127.0.98.62:37813 was elected leader, sending a full tablet report...
I20260927 02:44:19.811856 4961 ts_tablet_manager.cc:1428] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260927 02:44:19.812213 4810 heartbeater.cc:503] Master 127.0.98.62:37813 was elected leader, sending a full tablet report...
W20260927 02:44:19.820815 4678 tablet.cc:2369] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:44:19.975718 4944 tablet.cc:2369] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:19.985960 4968 raft_consensus.cc:482] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:19.986196 4968 raft_consensus.cc:504] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:19.987349 4968 leader_election.cc:288] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb69fc1360134804a11b4fbe18c6a3fc (127.0.98.3:40253), 2f4306545e814232b6788a60740bef1e (127.0.98.1:45065)
I20260927 02:44:19.992329 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.987680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59265 (local address 127.0.98.3:40253)
0927 02:44:19.987801 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.987808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.988478 (+ 670us) server_negotiation.cc:407] Connection header received
0927 02:44:19.988651 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.988654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.988711 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.988784 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:19.989098 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.989732 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.990956 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.991575 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.991681 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.991741 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.992048 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.992151 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:44:19.992434 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.988013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58191 (local address 127.0.98.1:45065)
0927 02:44:19.988138 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:19.988141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:19.988153 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:44:19.988224 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:19.988228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:19.988282 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:19.988368 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:19.988726 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.989465 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.991274 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.991861 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.991995 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.992050 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:19.992248 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 02:44:19.992315 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:44:19.992434 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.987630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45065 (local address 127.0.98.2:58191)
0927 02:44:19.987786 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:19.987804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:19.987929 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:19.988401 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:19.988407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:19.988435 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:19.988597 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.988607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.989585 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.989591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.991153 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.991167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.992005 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.992008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.992128 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.992145 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:44:19.992232 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:44:19.992312 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:44:19.992787 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:19.987639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40253 (local address 127.0.98.2:59265)
0927 02:44:19.988394 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:19.988408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:19.988543 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:19.988808 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:19.988813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:19.988837 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:19.988972 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.988980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.989842 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.989849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.990811 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:19.990819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:19.991702 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:19.991708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:19.991847 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:19.991911 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 02:44:19.992476 (+ 565us) client_negotiation.cc:240] Negotiation successful
0927 02:44:19.992583 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":375,"thread_start_us":135,"threads_started":1}
I20260927 02:44:19.993062 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74638802181849c090e11c47f229fae7" candidate_uuid: "96c9d6640afd45cc81940fe17cfb29b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4306545e814232b6788a60740bef1e" is_pre_election: true
I20260927 02:44:19.993356 4631 raft_consensus.cc:2454] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96c9d6640afd45cc81940fe17cfb29b2 in term 0.
I20260927 02:44:19.993592 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74638802181849c090e11c47f229fae7" candidate_uuid: "96c9d6640afd45cc81940fe17cfb29b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" is_pre_election: true
I20260927 02:44:19.993860 4697 leader_election.cc:302] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4306545e814232b6788a60740bef1e, 96c9d6640afd45cc81940fe17cfb29b2; no voters:
I20260927 02:44:19.993882 4897 raft_consensus.cc:2454] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96c9d6640afd45cc81940fe17cfb29b2 in term 0.
I20260927 02:44:19.994079 4968 raft_consensus.cc:2781] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:19.994161 4968 raft_consensus.cc:482] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:19.994240 4968 raft_consensus.cc:3037] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:19.995301 4968 raft_consensus.cc:504] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:19.995709 4968 leader_election.cc:288] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [CANDIDATE]: Term 1 election: Requested vote from peers bb69fc1360134804a11b4fbe18c6a3fc (127.0.98.3:40253), 2f4306545e814232b6788a60740bef1e (127.0.98.1:45065)
I20260927 02:44:19.995991 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74638802181849c090e11c47f229fae7" candidate_uuid: "96c9d6640afd45cc81940fe17cfb29b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb69fc1360134804a11b4fbe18c6a3fc"
I20260927 02:44:19.996114 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74638802181849c090e11c47f229fae7" candidate_uuid: "96c9d6640afd45cc81940fe17cfb29b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4306545e814232b6788a60740bef1e"
I20260927 02:44:19.996126 4897 raft_consensus.cc:3037] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:19.996232 4631 raft_consensus.cc:3037] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:19.997398 4631 raft_consensus.cc:2454] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96c9d6640afd45cc81940fe17cfb29b2 in term 1.
I20260927 02:44:19.997393 4897 raft_consensus.cc:2454] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96c9d6640afd45cc81940fe17cfb29b2 in term 1.
I20260927 02:44:19.997715 4697 leader_election.cc:302] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4306545e814232b6788a60740bef1e, 96c9d6640afd45cc81940fe17cfb29b2; no voters:
I20260927 02:44:19.997995 4968 raft_consensus.cc:2781] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:19.998203 4968 raft_consensus.cc:686] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 1 LEADER]: Becoming Leader. State: Replica: 96c9d6640afd45cc81940fe17cfb29b2, State: Running, Role: LEADER
I20260927 02:44:19.998509 4968 consensus_queue.cc:237] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } }
I20260927 02:44:20.001688 4487 catalog_manager.cc:5495] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96c9d6640afd45cc81940fe17cfb29b2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "96c9d6640afd45cc81940fe17cfb29b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96c9d6640afd45cc81940fe17cfb29b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:20.019769 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.016091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.0.98.2:43789)
0927 02:44:20.016218 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.016223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.016286 (+ 63us) server_negotiation.cc:407] Connection header received
0927 02:44:20.016470 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.016473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.016536 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.016662 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:20.016901 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:20.017402 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:20.018315 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:20.018545 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:20.018654 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:20.018953 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:20.019363 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:20.019492 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.019554 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.019613 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 02:44:20.023051 4811 tablet.cc:2369] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:20.025245 4631 raft_consensus.cc:1261] T 74638802181849c090e11c47f229fae7 P 2f4306545e814232b6788a60740bef1e [term 1 FOLLOWER]: Refusing update from remote peer 96c9d6640afd45cc81940fe17cfb29b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:20.025245 4897 raft_consensus.cc:1261] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 1 FOLLOWER]: Refusing update from remote peer 96c9d6640afd45cc81940fe17cfb29b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:20.026017 4970 consensus_queue.cc:1035] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb69fc1360134804a11b4fbe18c6a3fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:20.026240 4968 consensus_queue.cc:1035] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4306545e814232b6788a60740bef1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:20.044616 4484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:20.044803 4484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:20.047149 4484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a39c4a8f73d412baabc3baadbdcbdc7: Sending DeleteTablet for 3 replicas of tablet 74638802181849c090e11c47f229fae7
I20260927 02:44:20.048012 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet 74638802181849c090e11c47f229fae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55480
I20260927 02:44:20.048218 4744 tablet_service.cc:1514] Processing DeleteTablet for tablet 74638802181849c090e11c47f229fae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54232
I20260927 02:44:20.048514 4877 tablet_service.cc:1514] Processing DeleteTablet for tablet 74638802181849c090e11c47f229fae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57402
I20260927 02:44:20.048648 4984 tablet_replica.cc:330] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: stopping tablet replica
I20260927 02:44:20.048956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 4549
I20260927 02:44:20.048970 4985 tablet_replica.cc:330] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: stopping tablet replica
I20260927 02:44:20.048990 4984 raft_consensus.cc:2229] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:20.049804 4984 raft_consensus.cc:2258] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:20.050778 4985 raft_consensus.cc:2229] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:20.051102 4985 raft_consensus.cc:2258] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:20.051137 4984 ts_tablet_manager.cc:1905] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:20.052544 4985 ts_tablet_manager.cc:1905] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:20.053696 4984 ts_tablet_manager.cc:1918] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:20.053786 4984 log.cc:1196] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/74638802181849c090e11c47f229fae7
I20260927 02:44:20.054150 4984 ts_tablet_manager.cc:1939] T 74638802181849c090e11c47f229fae7 P 96c9d6640afd45cc81940fe17cfb29b2: Deleting consensus metadata
I20260927 02:44:20.055222 4985 ts_tablet_manager.cc:1918] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:20.055223 4472 catalog_manager.cc:4841] TS 96c9d6640afd45cc81940fe17cfb29b2 (127.0.98.2:43789): tablet 74638802181849c090e11c47f229fae7 (table test-table [id=9d2c4cd27dad40e5ba12c0cce10e9ea8]) successfully deleted
I20260927 02:44:20.055387 4985 log.cc:1196] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/74638802181849c090e11c47f229fae7
I20260927 02:44:20.056303 4985 ts_tablet_manager.cc:1939] T 74638802181849c090e11c47f229fae7 P bb69fc1360134804a11b4fbe18c6a3fc: Deleting consensus metadata
I20260927 02:44:20.057425 4472 catalog_manager.cc:4841] TS bb69fc1360134804a11b4fbe18c6a3fc (127.0.98.3:40253): tablet 74638802181849c090e11c47f229fae7 (table test-table [id=9d2c4cd27dad40e5ba12c0cce10e9ea8]) successfully deleted
W20260927 02:44:20.058822 4471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45065 (error 108)
I20260927 02:44:20.059638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 4682
I20260927 02:44:20.059801 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.059532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45065 (local address 127.0.0.1:55486)
0927 02:44:20.059683 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.059738 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 02:44:20.061179 4471 catalog_manager.cc:4584] TS 2f4306545e814232b6788a60740bef1e (127.0.98.1:45065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74638802181849c090e11c47f229fae7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45065: connect: Connection refused (error 111)
I20260927 02:44:20.066979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 4815
I20260927 02:44:20.073581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 4456
2026-09-27T02:44:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2353 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T02:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:20Z Disabled control of system clock
I20260927 02:44:20.103787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37385
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:44:20.209573 4990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:20.209798 4990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:20.209833 4990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:20.213682 4990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:20.213728 4990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:20.213744 4990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:20.213760 4990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:20.213775 4990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:20.218329 4990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:20.219250 4990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:20.220546 4990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:20.228389 4996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:20.228420 4990 server_base.cc:1034] running on GCE node
W20260927 02:44:20.228608 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:20.228484 4995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:20.229194 4990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:20.230458 4990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:20.231632 4990 hybrid_clock.cc:648] HybridClock initialized: now 1790477060231602 us; error 46 us; skew 500 ppm
I20260927 02:44:20.236639 4990 webserver.cc:460] Webserver started at http://127.0.98.62:44721/ using document root <none> and password file <none>
I20260927 02:44:20.237218 4990 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:20.237286 4990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:20.237543 4990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:20.239651 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "731f1912a13b4c68a4fabefd9912ac77"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "fd3eb9906a47ced7c0efc9dfaf84e994"
server_key_iv: "cb9c81ec8fa8018c66494d65adf47e88"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.240346 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "731f1912a13b4c68a4fabefd9912ac77"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "fd3eb9906a47ced7c0efc9dfaf84e994"
server_key_iv: "cb9c81ec8fa8018c66494d65adf47e88"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.244652 4990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:44:20.248342 5004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.249989 4990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:44:20.250115 4990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "731f1912a13b4c68a4fabefd9912ac77"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "fd3eb9906a47ced7c0efc9dfaf84e994"
server_key_iv: "cb9c81ec8fa8018c66494d65adf47e88"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.250260 4990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:20.255244 4990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:20.255932 4990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:20.256142 4990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:20.263166 4990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37385
I20260927 02:44:20.263173 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37385 every 8 connection(s)
I20260927 02:44:20.264149 4990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:20.267167 5057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:20.269213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 4990
I20260927 02:44:20.269361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:20.269630 392 external_mini_cluster.cc:1428] Setting key d71493ba406de4fdeac5e3f585aec3be
I20260927 02:44:20.272831 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77: Bootstrap starting.
I20260927 02:44:20.275107 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.271051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.0.98.62:37385)
0927 02:44:20.271648 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.271665 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.271699 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:44:20.272368 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.272387 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.272734 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.273075 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.273891 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.273896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.273907 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.273918 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.273972 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.273989 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.274102 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.274202 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.274448 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":136,"threads_started":1}
I20260927 02:44:20.275172 5057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:20.275942 5057 log.cc:824] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:20.277834 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77: No bootstrap required, opened a new log
I20260927 02:44:20.281342 5057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731f1912a13b4c68a4fabefd9912ac77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37385 } }
I20260927 02:44:20.281566 5057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:20.281641 5057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731f1912a13b4c68a4fabefd9912ac77, State: Initialized, Role: FOLLOWER
I20260927 02:44:20.282035 5057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731f1912a13b4c68a4fabefd9912ac77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37385 } }
I20260927 02:44:20.282181 5057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:20.282263 5057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:20.282382 5057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:20.283295 5057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731f1912a13b4c68a4fabefd9912ac77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37385 } }
I20260927 02:44:20.283564 5057 leader_election.cc:302] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 731f1912a13b4c68a4fabefd9912ac77; no voters:
I20260927 02:44:20.283783 5057 leader_election.cc:288] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:20.283986 5062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:20.284232 5062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [term 1 LEADER]: Becoming Leader. State: Replica: 731f1912a13b4c68a4fabefd9912ac77, State: Running, Role: LEADER
I20260927 02:44:20.284533 5062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731f1912a13b4c68a4fabefd9912ac77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37385 } }
I20260927 02:44:20.284966 5057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:20.286113 5064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 731f1912a13b4c68a4fabefd9912ac77. Latest consensus state: current_term: 1 leader_uuid: "731f1912a13b4c68a4fabefd9912ac77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731f1912a13b4c68a4fabefd9912ac77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37385 } } }
I20260927 02:44:20.286244 5064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:20.286556 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "731f1912a13b4c68a4fabefd9912ac77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731f1912a13b4c68a4fabefd9912ac77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37385 } } }
I20260927 02:44:20.286665 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:20.286753 5071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:20.289474 5071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:20.294512 5071 catalog_manager.cc:1347] Generated new cluster ID: 01961f8d2c484eda858ed285670c5788
I20260927 02:44:20.294576 5071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:20.303428 5071 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:20.303709 5071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:20.312536 5071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77: Generated new TSK 0
I20260927 02:44:20.313213 5071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:20.321744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37385
--builtin_ntp_servers=127.0.98.20:42853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:44:20.427515 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:20.427759 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:20.427836 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:20.431782 5081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:20.431830 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:20.431953 5081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:20.437193 5081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:20.438195 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:20.439520 5081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:44:20.448783 5081 server_base.cc:1034] running on GCE node
W20260927 02:44:20.448750 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:20.448750 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:20.448774 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:20.449376 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:20.450026 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:20.451215 5081 hybrid_clock.cc:648] HybridClock initialized: now 1790477060451200 us; error 27 us; skew 500 ppm
I20260927 02:44:20.453058 5081 webserver.cc:460] Webserver started at http://127.0.98.1:42359/ using document root <none> and password file <none>
I20260927 02:44:20.453668 5081 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:20.453759 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:20.453981 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:20.456027 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "22e6670c96bf40b5811011a3cd4a0de4"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "8745277c5b4d32c7c19393ae66bdc23b"
server_key_iv: "f8edcaaa907554a2428815aa0ce28bab"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.456580 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22e6670c96bf40b5811011a3cd4a0de4"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "8745277c5b4d32c7c19393ae66bdc23b"
server_key_iv: "f8edcaaa907554a2428815aa0ce28bab"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.460042 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:20.462337 5095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.463515 5081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:20.463656 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "22e6670c96bf40b5811011a3cd4a0de4"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "8745277c5b4d32c7c19393ae66bdc23b"
server_key_iv: "f8edcaaa907554a2428815aa0ce28bab"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.463763 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:20.468107 5081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:20.468665 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:20.468844 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:20.469352 5081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:20.470183 5081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:20.470252 5081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.470316 5081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:20.470367 5081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.478968 5081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40787
I20260927 02:44:20.479010 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40787 every 8 connection(s)
I20260927 02:44:20.479771 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:20.485314 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.481413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39309 (local address 127.0.98.62:37385)
0927 02:44:20.481564 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.481569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.482550 (+ 981us) server_negotiation.cc:407] Connection header received
0927 02:44:20.483720 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.483722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.483757 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.483830 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.484884 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.484888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.484889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.484899 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.484916 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.484924 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.485003 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.485131 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.485166 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:44:20.486172 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.481823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37385 (local address 127.0.98.1:39309)
0927 02:44:20.482415 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.482452 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:20.483545 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:20.484062 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:20.484068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:20.484149 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:20.484652 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:20.484654 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:20.484695 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:20.484754 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:20.485017 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:20.485023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:20.485027 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:20.485148 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 02:44:20.485464 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260927 02:44:20.487274 5209 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37385
I20260927 02:44:20.487327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 5081
I20260927 02:44:20.487430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:20.487510 5209 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:20.487735 392 external_mini_cluster.cc:1428] Setting key ad6f0d56716718edebb9b9844c97e811
I20260927 02:44:20.487950 5209 heartbeater.cc:511] Master 127.0.98.62:37385 requested a full tablet report, sending...
I20260927 02:44:20.489486 5021 ts_manager.cc:194] Registered new tserver with Master: 22e6670c96bf40b5811011a3cd4a0de4 (127.0.98.1:40787)
I20260927 02:44:20.489737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37385
--builtin_ntp_servers=127.0.98.20:42853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:44:20.593698 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:20.593940 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:20.593984 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:20.597988 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:20.598033 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:20.598086 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:20.602639 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:20.603585 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:20.604883 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:20.614061 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:20.614066 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:20.614109 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:20.614297 5212 server_base.cc:1034] running on GCE node
I20260927 02:44:20.614737 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:20.615268 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:20.616421 5212 hybrid_clock.cc:648] HybridClock initialized: now 1790477060616407 us; error 26 us; skew 500 ppm
I20260927 02:44:20.618113 5212 webserver.cc:460] Webserver started at http://127.0.98.2:37997/ using document root <none> and password file <none>
I20260927 02:44:20.618654 5212 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:20.618712 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:20.618938 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:20.620950 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "90244616ec9e438ca64af729ee358a10"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "aba686be159b7d7e23025aab62c08e8b"
server_key_iv: "d00b20f95c96d879602a36dc80a10c39"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.621496 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "90244616ec9e438ca64af729ee358a10"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "aba686be159b7d7e23025aab62c08e8b"
server_key_iv: "d00b20f95c96d879602a36dc80a10c39"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.625020 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 02:44:20.627463 5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.628742 5212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:20.628859 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "90244616ec9e438ca64af729ee358a10"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "aba686be159b7d7e23025aab62c08e8b"
server_key_iv: "d00b20f95c96d879602a36dc80a10c39"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.628957 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:20.633240 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:20.633776 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:20.633940 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:20.634433 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:20.635282 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:20.635331 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.635409 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:20.635439 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.643572 5212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45943
I20260927 02:44:20.643637 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45943 every 8 connection(s)
I20260927 02:44:20.644444 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:20.645009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 5212
I20260927 02:44:20.645139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:20.645447 392 external_mini_cluster.cc:1428] Setting key 818cac943fb157540928708148eaa4a1
I20260927 02:44:20.647620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37385
--builtin_ntp_servers=127.0.98.20:42853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 02:44:20.651218 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.646329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41261 (local address 127.0.98.62:37385)
0927 02:44:20.646465 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.646470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.647736 (+ 1266us) server_negotiation.cc:407] Connection header received
0927 02:44:20.649529 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.649533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.649572 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.649648 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.650703 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.650710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.650716 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.650732 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.650757 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.650770 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.650867 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.650993 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.651038 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:44:20.651945 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.646788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37385 (local address 127.0.98.2:41261)
0927 02:44:20.647562 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.647628 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:20.649318 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:20.649799 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:20.649805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:20.649884 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:20.650451 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:20.650453 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:20.650489 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:20.650554 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:20.650858 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:20.650863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:20.650867 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:20.650996 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 02:44:20.651163 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260927 02:44:20.653050 5340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37385
I20260927 02:44:20.653267 5340 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:20.653611 5340 heartbeater.cc:511] Master 127.0.98.62:37385 requested a full tablet report, sending...
I20260927 02:44:20.654618 5021 ts_manager.cc:194] Registered new tserver with Master: 90244616ec9e438ca64af729ee358a10 (127.0.98.2:45943)
W20260927 02:44:20.753265 5343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:20.753492 5343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:20.753557 5343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:20.757639 5343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:20.757685 5343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:20.757768 5343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:20.762279 5343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:20.763263 5343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:20.764577 5343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:20.773597 5348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:20.773639 5351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:20.773600 5349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:20.773952 5343 server_base.cc:1034] running on GCE node
I20260927 02:44:20.774426 5343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:20.774996 5343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:20.776172 5343 hybrid_clock.cc:648] HybridClock initialized: now 1790477060776145 us; error 41 us; skew 500 ppm
I20260927 02:44:20.777854 5343 webserver.cc:460] Webserver started at http://127.0.98.3:44853/ using document root <none> and password file <none>
I20260927 02:44:20.778419 5343 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:20.778513 5343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:20.778726 5343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:20.780656 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "9787057b9a7044c18d7cf2f1fa173d54"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "0bfcce03a7c9410153917262f94c6ba9"
server_key_iv: "8a61a90b4b95a3adc0eb5d852ed55ccb"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.781188 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9787057b9a7044c18d7cf2f1fa173d54"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "0bfcce03a7c9410153917262f94c6ba9"
server_key_iv: "8a61a90b4b95a3adc0eb5d852ed55ccb"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.784549 5343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:20.787034 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.788370 5343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:20.788504 5343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "9787057b9a7044c18d7cf2f1fa173d54"
format_stamp: "Formatted at 2026-09-27 02:44:20 on dist-test-slave-gkrl"
server_key: "0bfcce03a7c9410153917262f94c6ba9"
server_key_iv: "8a61a90b4b95a3adc0eb5d852ed55ccb"
server_key_version: "encryptionkey@0"
I20260927 02:44:20.788611 5343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:20.792945 5343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:20.793500 5343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:20.793674 5343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:20.794198 5343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:20.795084 5343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:20.795156 5343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.795222 5343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:20.795270 5343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:20.804062 5343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34073
I20260927 02:44:20.804097 5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34073 every 8 connection(s)
I20260927 02:44:20.804895 5343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:20.810740 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.806625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44397 (local address 127.0.98.62:37385)
0927 02:44:20.806758 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.806763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.807943 (+ 1180us) server_negotiation.cc:407] Connection header received
0927 02:44:20.809169 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.809172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.809209 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.809285 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.810338 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.810341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.810342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.810353 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.810371 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.810380 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.810459 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.810607 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.810641 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:20.811522 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.807058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37385 (local address 127.0.98.3:44397)
0927 02:44:20.807748 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.807785 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:20.808980 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:20.809428 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:20.809434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:20.809513 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:20.810028 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:20.810029 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:20.810087 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:20.810171 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:20.810466 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:20.810473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:20.810479 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:44:20.810638 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 02:44:20.810836 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260927 02:44:20.812608 5471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37385
I20260927 02:44:20.812824 5471 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:20.813197 5471 heartbeater.cc:511] Master 127.0.98.62:37385 requested a full tablet report, sending...
I20260927 02:44:20.813941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 5343
I20260927 02:44:20.814090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:20.814138 5021 ts_manager.cc:194] Registered new tserver with Master: 9787057b9a7044c18d7cf2f1fa173d54 (127.0.98.3:34073)
I20260927 02:44:20.814368 392 external_mini_cluster.cc:1428] Setting key 21d6e4298de36b2b79bb5848d3664183
I20260927 02:44:20.816226 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:20.819490 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.817881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.0.98.62:37385)
0927 02:44:20.818035 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.818038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.818185 (+ 147us) server_negotiation.cc:407] Connection header received
0927 02:44:20.818322 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.818325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.818362 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.818449 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.819135 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.819138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.819141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.819152 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.819169 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.819177 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.819272 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.819341 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.819379 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:44:20.822024 5021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:20.824867 5021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:20.835618 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.833089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34073 (local address 127.0.0.1:56860)
0927 02:44:20.833284 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.833300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:20.833379 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:20.834271 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:20.834276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:20.834294 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:20.834787 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:20.834789 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:20.834800 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:20.834824 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:20.835441 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:20.835443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:20.835447 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:20.835495 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:44:20.835519 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260927 02:44:20.835655 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.833203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.0.98.2:45943)
0927 02:44:20.833939 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.834050 (+ 111us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.834073 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:44:20.834125 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.834130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.834292 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.834426 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.835079 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.835084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.835094 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.835104 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.835141 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.835164 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.835271 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.835414 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.835485 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":90,"threads_started":1}
I20260927 02:44:20.835799 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.832440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40787 (local address 127.0.0.1:49488)
0927 02:44:20.832769 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.832815 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:20.832909 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:20.834068 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:20.834073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:20.834138 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:20.834633 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:20.834635 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:20.834652 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:20.834715 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:20.835520 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:20.835523 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:20.835527 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:20.835586 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:44:20.835617 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":63,"threads_started":1}
I20260927 02:44:20.835618 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.832893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45943 (local address 127.0.0.1:50126)
0927 02:44:20.833273 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.833288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:20.833360 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:20.834406 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:20.834411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:20.834433 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:20.834947 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:20.834951 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:20.834960 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:20.834983 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:20.835260 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:20.835268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:20.835273 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:44:20.835391 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 02:44:20.835485 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":95,"threads_started":1}
I20260927 02:44:20.836236 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.832549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.98.1:40787)
0927 02:44:20.833613 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.833626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.833655 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:44:20.833710 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.833716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.833895 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.834081 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.835288 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.835295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.835305 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.835315 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.835376 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.835401 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.835941 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.835995 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.836065 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":127,"threads_started":1}
I20260927 02:44:20.837024 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.833528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.0.98.3:34073)
0927 02:44:20.833855 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.833868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.833893 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:44:20.833952 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.833958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.834125 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.834467 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.835261 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.835266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.835275 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.835284 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.835319 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.835343 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.836051 (+ 708us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.836096 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.836869 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":142,"threads_started":1}
I20260927 02:44:20.838619 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc170cbc2734dfe9fb797920a5b9992 (DEFAULT_TABLE table=test-table [id=d5d13a887a7e4a1bba2f1f8de95ccd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:20.838704 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc170cbc2734dfe9fb797920a5b9992 (DEFAULT_TABLE table=test-table [id=d5d13a887a7e4a1bba2f1f8de95ccd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:20.839627 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc170cbc2734dfe9fb797920a5b9992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:20.839670 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc170cbc2734dfe9fb797920a5b9992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:20.839406 5405 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc170cbc2734dfe9fb797920a5b9992 (DEFAULT_TABLE table=test-table [id=d5d13a887a7e4a1bba2f1f8de95ccd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:20.840270 5405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc170cbc2734dfe9fb797920a5b9992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:20.845263 5488 tablet_bootstrap.cc:492] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4: Bootstrap starting.
I20260927 02:44:20.846760 5489 tablet_bootstrap.cc:492] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: Bootstrap starting.
I20260927 02:44:20.847074 5490 tablet_bootstrap.cc:492] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10: Bootstrap starting.
I20260927 02:44:20.847435 5488 tablet_bootstrap.cc:654] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:20.848353 5488 log.cc:824] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:20.849040 5490 tablet_bootstrap.cc:654] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:20.849040 5489 tablet_bootstrap.cc:654] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:20.849817 5489 log.cc:824] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:20.849817 5490 log.cc:824] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:20.850231 5488 tablet_bootstrap.cc:492] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4: No bootstrap required, opened a new log
I20260927 02:44:20.850418 5488 ts_tablet_manager.cc:1397] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:44:20.851588 5489 tablet_bootstrap.cc:492] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: No bootstrap required, opened a new log
I20260927 02:44:20.851758 5489 ts_tablet_manager.cc:1397] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:44:20.851861 5490 tablet_bootstrap.cc:492] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10: No bootstrap required, opened a new log
I20260927 02:44:20.852054 5490 ts_tablet_manager.cc:1397] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 02:44:20.854106 5488 raft_consensus.cc:348] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.854344 5488 raft_consensus.cc:374] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:20.854394 5488 raft_consensus.cc:729] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e6670c96bf40b5811011a3cd4a0de4, State: Initialized, Role: FOLLOWER
I20260927 02:44:20.854820 5488 consensus_queue.cc:260] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.855593 5488 ts_tablet_manager.cc:1428] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:44:20.855964 5209 heartbeater.cc:503] Master 127.0.98.62:37385 was elected leader, sending a full tablet report...
I20260927 02:44:20.856241 5489 raft_consensus.cc:348] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.856287 5490 raft_consensus.cc:348] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.856468 5489 raft_consensus.cc:374] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:20.856479 5490 raft_consensus.cc:374] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:20.856523 5490 raft_consensus.cc:729] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90244616ec9e438ca64af729ee358a10, State: Initialized, Role: FOLLOWER
I20260927 02:44:20.856519 5489 raft_consensus.cc:729] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9787057b9a7044c18d7cf2f1fa173d54, State: Initialized, Role: FOLLOWER
I20260927 02:44:20.856956 5489 consensus_queue.cc:260] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.856950 5490 consensus_queue.cc:260] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.857717 5471 heartbeater.cc:503] Master 127.0.98.62:37385 was elected leader, sending a full tablet report...
I20260927 02:44:20.857829 5489 ts_tablet_manager.cc:1428] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:44:20.857829 5490 ts_tablet_manager.cc:1428] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 02:44:20.858721 5340 heartbeater.cc:503] Master 127.0.98.62:37385 was elected leader, sending a full tablet report...
W20260927 02:44:20.895843 5341 tablet.cc:2369] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:20.966714 5495 raft_consensus.cc:482] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:20.967026 5495 raft_consensus.cc:504] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.968283 5495 leader_election.cc:288] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22e6670c96bf40b5811011a3cd4a0de4 (127.0.98.1:40787), 90244616ec9e438ca64af729ee358a10 (127.0.98.2:45943)
I20260927 02:44:20.970233 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.968563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45943 (local address 127.0.98.3:48475)
0927 02:44:20.968713 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.968731 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:20.968820 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:20.969143 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:20.969146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:20.969158 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:20.969667 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:20.969670 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:20.969684 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:20.969723 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:20.969978 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:20.969981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:20.969984 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:44:20.970065 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:44:20.970099 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 02:44:20.970305 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.968662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48475 (local address 127.0.98.2:45943)
0927 02:44:20.968813 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.968818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.968838 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:44:20.968969 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.968973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.969018 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.969084 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.969853 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.969856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.969858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.969868 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.969886 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.969894 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.969979 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.970088 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.970129 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:44:20.970721 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.968563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40787 (local address 127.0.98.3:52815)
0927 02:44:20.969043 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:20.969055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:20.969165 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:20.969399 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:20.969401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:20.969416 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:20.970069 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:20.970071 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:20.970081 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:20.970108 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:20.970472 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:20.970475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:20.970478 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:44:20.970557 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:44:20.970589 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":109,"thread_start_us":128,"threads_started":1}
I20260927 02:44:20.970719 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:20.968813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52815 (local address 127.0.98.1:40787)
0927 02:44:20.968941 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:20.968945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:20.969172 (+ 227us) server_negotiation.cc:407] Connection header received
0927 02:44:20.969242 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:20.969244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:20.969273 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:20.969365 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:20.970267 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:20.970272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:20.970276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:20.970291 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:20.970311 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:20.970321 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:20.970431 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:20.970576 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:44:20.970612 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:44:20.970870 5294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc170cbc2734dfe9fb797920a5b9992" candidate_uuid: "9787057b9a7044c18d7cf2f1fa173d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90244616ec9e438ca64af729ee358a10" is_pre_election: true
I20260927 02:44:20.971176 5294 raft_consensus.cc:2454] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9787057b9a7044c18d7cf2f1fa173d54 in term 0.
I20260927 02:44:20.971375 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc170cbc2734dfe9fb797920a5b9992" candidate_uuid: "9787057b9a7044c18d7cf2f1fa173d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22e6670c96bf40b5811011a3cd4a0de4" is_pre_election: true
I20260927 02:44:20.971639 5360 leader_election.cc:302] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90244616ec9e438ca64af729ee358a10, 9787057b9a7044c18d7cf2f1fa173d54; no voters:
I20260927 02:44:20.971663 5163 raft_consensus.cc:2454] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9787057b9a7044c18d7cf2f1fa173d54 in term 0.
I20260927 02:44:20.971923 5495 raft_consensus.cc:2781] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:20.972015 5495 raft_consensus.cc:482] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:20.972075 5495 raft_consensus.cc:3037] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:20.973057 5495 raft_consensus.cc:504] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.973392 5495 leader_election.cc:288] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [CANDIDATE]: Term 1 election: Requested vote from peers 22e6670c96bf40b5811011a3cd4a0de4 (127.0.98.1:40787), 90244616ec9e438ca64af729ee358a10 (127.0.98.2:45943)
I20260927 02:44:20.973739 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc170cbc2734dfe9fb797920a5b9992" candidate_uuid: "9787057b9a7044c18d7cf2f1fa173d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22e6670c96bf40b5811011a3cd4a0de4"
I20260927 02:44:20.973745 5294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc170cbc2734dfe9fb797920a5b9992" candidate_uuid: "9787057b9a7044c18d7cf2f1fa173d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90244616ec9e438ca64af729ee358a10"
I20260927 02:44:20.973883 5294 raft_consensus.cc:3037] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:20.973873 5163 raft_consensus.cc:3037] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:20.974879 5294 raft_consensus.cc:2454] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9787057b9a7044c18d7cf2f1fa173d54 in term 1.
I20260927 02:44:20.975003 5163 raft_consensus.cc:2454] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9787057b9a7044c18d7cf2f1fa173d54 in term 1.
I20260927 02:44:20.975147 5360 leader_election.cc:302] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90244616ec9e438ca64af729ee358a10, 9787057b9a7044c18d7cf2f1fa173d54; no voters:
I20260927 02:44:20.975324 5495 raft_consensus.cc:2781] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:20.975555 5495 raft_consensus.cc:686] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 1 LEADER]: Becoming Leader. State: Replica: 9787057b9a7044c18d7cf2f1fa173d54, State: Running, Role: LEADER
I20260927 02:44:20.975870 5495 consensus_queue.cc:237] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } }
I20260927 02:44:20.978873 5015 catalog_manager.cc:5495] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9787057b9a7044c18d7cf2f1fa173d54 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9787057b9a7044c18d7cf2f1fa173d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9787057b9a7044c18d7cf2f1fa173d54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 } health_report { overall_health: UNKNOWN } } }
W20260927 02:44:20.980878 5210 tablet.cc:2369] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:21.023839 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.022303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.0.98.3:34073)
0927 02:44:21.022443 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:21.022448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:21.022462 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:21.022571 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:21.022573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:21.022631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:21.022724 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:21.023429 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:21.023432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:21.023435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:21.023446 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:21.023477 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:21.023492 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:21.023587 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:21.023653 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:44:21.023694 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:44:21.029479 5294 raft_consensus.cc:1261] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 1 FOLLOWER]: Refusing update from remote peer 9787057b9a7044c18d7cf2f1fa173d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:21.029479 5163 raft_consensus.cc:1261] T 4cc170cbc2734dfe9fb797920a5b9992 P 22e6670c96bf40b5811011a3cd4a0de4 [term 1 FOLLOWER]: Refusing update from remote peer 9787057b9a7044c18d7cf2f1fa173d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:21.030148 5495 consensus_queue.cc:1035] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22e6670c96bf40b5811011a3cd4a0de4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:21.030304 5498 consensus_queue.cc:1035] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90244616ec9e438ca64af729ee358a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:21.037353 5503 mvcc.cc:205] Tried to move back new op lower bound from 7333794041969864704 to 7333794041759629312. Current Snapshot: MvccSnapshot[applied={T|T < 7333794041969864704}]
I20260927 02:44:21.039471 5502 mvcc.cc:205] Tried to move back new op lower bound from 7333794041969864704 to 7333794041759629312. Current Snapshot: MvccSnapshot[applied={T|T < 7333794041969864704}]
I20260927 02:44:21.045401 5021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:21.045562 5021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:21.047959 5021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 731f1912a13b4c68a4fabefd9912ac77: Sending DeleteTablet for 3 replicas of tablet 4cc170cbc2734dfe9fb797920a5b9992
I20260927 02:44:21.048678 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc170cbc2734dfe9fb797920a5b9992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:21 UTC) from {username='slave'} at 127.0.0.1:50126
I20260927 02:44:21.048709 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc170cbc2734dfe9fb797920a5b9992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:21 UTC) from {username='slave'} at 127.0.0.1:49488
I20260927 02:44:21.049167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 5081
I20260927 02:44:21.049253 5405 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc170cbc2734dfe9fb797920a5b9992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:21 UTC) from {username='slave'} at 127.0.0.1:56860
I20260927 02:44:21.049755 5513 tablet_replica.cc:330] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: stopping tablet replica
I20260927 02:44:21.050127 5513 raft_consensus.cc:2229] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:21.050450 5513 raft_consensus.cc:2258] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:21.050800 5511 tablet_replica.cc:330] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10: stopping tablet replica
I20260927 02:44:21.051081 5511 raft_consensus.cc:2229] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:21.051352 5511 raft_consensus.cc:2258] T 4cc170cbc2734dfe9fb797920a5b9992 P 90244616ec9e438ca64af729ee358a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:21.052106 5513 ts_tablet_manager.cc:1905] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:21.055004 5513 ts_tablet_manager.cc:1918] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:21.055143 5513 log.cc:1196] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/4cc170cbc2734dfe9fb797920a5b9992
I20260927 02:44:21.055677 5513 ts_tablet_manager.cc:1939] T 4cc170cbc2734dfe9fb797920a5b9992 P 9787057b9a7044c18d7cf2f1fa173d54: Deleting consensus metadata
W20260927 02:44:21.056833 5008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40787 (error 108)
I20260927 02:44:21.056969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 5212
I20260927 02:44:21.058676 5008 catalog_manager.cc:4841] TS 9787057b9a7044c18d7cf2f1fa173d54 (127.0.98.3:34073): tablet 4cc170cbc2734dfe9fb797920a5b9992 (table test-table [id=d5d13a887a7e4a1bba2f1f8de95ccd7e]) successfully deleted
I20260927 02:44:21.059585 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.059245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40787 (local address 127.0.0.1:49492)
0927 02:44:21.059409 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:21.059489 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 02:44:21.059755 5008 catalog_manager.cc:4584] TS 22e6670c96bf40b5811011a3cd4a0de4 (127.0.98.1:40787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cc170cbc2734dfe9fb797920a5b9992: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40787: connect: Connection refused (error 111)
I20260927 02:44:21.062309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 5343
I20260927 02:44:21.064081 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.063808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45943 (local address 127.0.0.1:50134)
0927 02:44:21.063968 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:21.064017 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:44:21.067799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 4990
2026-09-27T02:44:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (994 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 02:44:21.090093 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:44:21.091630 392 webserver.cc:460] Webserver started at https://127.0.0.1:40253/ using document root <none> and password file <none>
I20260927 02:44:21.091806 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:44:21.092128 392 webserver.cc:460] Webserver started at http://127.0.0.1:46239/ using document root <none> and password file <none>
2026-09-27T02:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:21Z Disabled control of system clock
I20260927 02:44:21.100693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43001
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43001
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40253/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/testchainca.cert with env {}
W20260927 02:44:21.205459 5520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:21.205683 5520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:21.205721 5520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:21.209806 5520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:21.209859 5520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:21.209875 5520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:21.209892 5520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:21.209906 5520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:44:21.209921 5520 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:44:21.209934 5520 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40253/jwks.json
I20260927 02:44:21.214962 5520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40253/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:21.215787 5520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:21.216728 5520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:21.224525 5528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:21.224653 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:21.224577 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:21.224548 5520 server_base.cc:1034] running on GCE node
I20260927 02:44:21.225266 5520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:21.226244 5520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:21.227418 5520 hybrid_clock.cc:648] HybridClock initialized: now 1790477061227395 us; error 34 us; skew 500 ppm
I20260927 02:44:21.229403 5520 webserver.cc:460] Webserver started at http://127.0.98.62:37421/ using document root <none> and password file <none>
I20260927 02:44:21.229945 5520 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:21.230010 5520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:21.230208 5520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:21.232161 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "d0d29fb82e0949f88c6a34c309d503f5"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "e5c4225285d1f0da01a0e2f771d656a2"
server_key_iv: "c87b21b5a338b5ce10b0ecab47af762f"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.232705 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0d29fb82e0949f88c6a34c309d503f5"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "e5c4225285d1f0da01a0e2f771d656a2"
server_key_iv: "c87b21b5a338b5ce10b0ecab47af762f"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.236269 5520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:21.238648 5534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:21.240005 5520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:21.240145 5520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "d0d29fb82e0949f88c6a34c309d503f5"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "e5c4225285d1f0da01a0e2f771d656a2"
server_key_iv: "c87b21b5a338b5ce10b0ecab47af762f"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.240259 5520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:21.253255 5520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:21.253854 5520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:21.254047 5520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:21.261348 5520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43001
I20260927 02:44:21.261374 5586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43001 every 8 connection(s)
I20260927 02:44:21.262231 5520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:21.265393 5587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:21.265751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 5520
I20260927 02:44:21.265861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:21.266134 392 external_mini_cluster.cc:1428] Setting key cfee0878affbdaf02b8ac8dd5bfc7c88
I20260927 02:44:21.272780 5587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5: Bootstrap starting.
I20260927 02:44:21.274998 5587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:21.275831 5587 log.cc:824] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:21.276144 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.267454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.0.98.62:43001)
0927 02:44:21.268182 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:21.268202 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:21.268248 (+ 46us) server_negotiation.cc:407] Connection header received
0927 02:44:21.269019 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:21.269042 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:21.269557 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:21.269927 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:21.270378 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.272044 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.272948 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.273423 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.273601 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.274486 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:21.274491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:21.274502 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:21.274854 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:21.274904 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:21.274922 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:21.275040 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:21.275157 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:44:21.275391 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260927 02:44:21.278047 5587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5: No bootstrap required, opened a new log
I20260927 02:44:21.278964 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:44:21.282368 5587 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d29fb82e0949f88c6a34c309d503f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43001 } }
I20260927 02:44:21.282609 5587 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:21.282672 5587 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d29fb82e0949f88c6a34c309d503f5, State: Initialized, Role: FOLLOWER
I20260927 02:44:21.283195 5587 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d29fb82e0949f88c6a34c309d503f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43001 } }
I20260927 02:44:21.283365 5587 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:21.283437 5587 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:21.283538 5587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:21.284521 5587 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d29fb82e0949f88c6a34c309d503f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43001 } }
I20260927 02:44:21.284871 5587 leader_election.cc:302] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d29fb82e0949f88c6a34c309d503f5; no voters:
I20260927 02:44:21.285179 5587 leader_election.cc:288] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:21.285288 5592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:21.285496 5592 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [term 1 LEADER]: Becoming Leader. State: Replica: d0d29fb82e0949f88c6a34c309d503f5, State: Running, Role: LEADER
I20260927 02:44:21.285806 5592 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d29fb82e0949f88c6a34c309d503f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43001 } }
I20260927 02:44:21.286511 5587 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:21.287551 5594 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d29fb82e0949f88c6a34c309d503f5. Latest consensus state: current_term: 1 leader_uuid: "d0d29fb82e0949f88c6a34c309d503f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d29fb82e0949f88c6a34c309d503f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43001 } } }
I20260927 02:44:21.287712 5594 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:21.288115 5601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:21.287997 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d29fb82e0949f88c6a34c309d503f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d29fb82e0949f88c6a34c309d503f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43001 } } }
I20260927 02:44:21.288763 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:21.291235 5601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:21.296339 5601 catalog_manager.cc:1347] Generated new cluster ID: 427fb53999414a74ba5059a1eec1b1dd
I20260927 02:44:21.296401 5601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:21.309473 5601 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:21.310730 5601 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:21.320906 5601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d29fb82e0949f88c6a34c309d503f5: Generated new TSK 0
I20260927 02:44:21.321576 5601 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:44:21.354992 5617 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 02:44:21.355105 5617 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:44:21.355554 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.339096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.0.98.62:43001)
0927 02:44:21.339232 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:21.339237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:21.339463 (+ 226us) server_negotiation.cc:407] Connection header received
0927 02:44:21.339663 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:21.339665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:21.339710 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:21.339794 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:44:21.340103 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.340632 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.341878 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.342098 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.342194 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.342534 (+ 340us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:44:21.355239 (+ 12705us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:44:21.355453 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 02:44:21.355629 5590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 02:44:21.357089 5616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:21.339037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43001 (local address 127.0.0.1:46700)
0927 02:44:21.339374 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:21.339389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:21.339518 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:21.339830 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:21.339833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:21.339842 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:44:21.339994 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:21.340001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.340735 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.340738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:21.341752 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:21.341762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.342211 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.342229 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:21.342323 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.342372 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:44:21.355443 (+ 13071us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:44:21.355671 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 02:44:21.357829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 5520
2026-09-27T02:44:21Z chronyd exiting
I20260927 02:44:21.394490 392 test_util.cc:175] -----------------------------------------------
I20260927 02:44:21.394593 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477031916752-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 02:44:21.395602 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:44:21.396503 392 webserver.cc:460] Webserver started at https://127.0.0.1:43317/ using document root <none> and password file <none>
I20260927 02:44:21.396628 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:44:21.396885 392 webserver.cc:460] Webserver started at http://127.0.0.1:35055/ using document root <none> and password file <none>
2026-09-27T02:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:21Z Disabled control of system clock
I20260927 02:44:21.405237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42143
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42143
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43317/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/testchainca.cert with env {}
W20260927 02:44:21.510574 5623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:21.510797 5623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:21.510833 5623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:21.514789 5623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:21.514840 5623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:21.514855 5623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:21.514873 5623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:21.514887 5623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:44:21.514945 5623 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:44:21.514966 5623 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43317/jwks.json
I20260927 02:44:21.520134 5623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43317/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:21.521066 5623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:21.522002 5623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:21.530236 5631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:21.530253 5628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:21.530292 5623 server_base.cc:1034] running on GCE node
W20260927 02:44:21.530247 5629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:21.530990 5623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:21.532096 5623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:21.533264 5623 hybrid_clock.cc:648] HybridClock initialized: now 1790477061533232 us; error 47 us; skew 500 ppm
I20260927 02:44:21.535092 5623 webserver.cc:460] Webserver started at http://127.0.98.62:44207/ using document root <none> and password file <none>
I20260927 02:44:21.535637 5623 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:21.535696 5623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:21.535954 5623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:21.537905 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "b9af03e3a9c6487892664720ffec2a7c"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "7260622b981a36b70e04e4946c0b3263"
server_key_iv: "29c0600bd694e57961bf5151df898a78"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.538460 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9af03e3a9c6487892664720ffec2a7c"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "7260622b981a36b70e04e4946c0b3263"
server_key_iv: "29c0600bd694e57961bf5151df898a78"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.541986 5623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:44:21.544457 5637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:21.545681 5623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:21.545809 5623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "b9af03e3a9c6487892664720ffec2a7c"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "7260622b981a36b70e04e4946c0b3263"
server_key_iv: "29c0600bd694e57961bf5151df898a78"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.545910 5623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:21.567116 5623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:21.567764 5623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:21.567986 5623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:21.574841 5623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42143
I20260927 02:44:21.575079 5689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42143 every 8 connection(s)
I20260927 02:44:21.575806 5623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:21.578807 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:21.580435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 5623
I20260927 02:44:21.580581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:21.580868 392 external_mini_cluster.cc:1428] Setting key 584a4801b2301c9d242ecebe46211849
I20260927 02:44:21.585095 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c: Bootstrap starting.
I20260927 02:44:21.587637 5690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:21.588536 5690 log.cc:824] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:21.589996 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.582190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.98.62:42143)
0927 02:44:21.582752 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:21.582768 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:21.582801 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:44:21.583427 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:21.583443 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:21.583811 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:21.584139 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:21.584458 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.585434 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.586600 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.587058 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.587170 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.588146 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:21.588152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:21.588166 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:21.588657 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:21.588721 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:21.588748 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:21.588852 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:21.589102 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:44:21.589338 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":120,"threads_started":1}
I20260927 02:44:21.590456 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c: No bootstrap required, opened a new log
I20260927 02:44:21.592849 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:44:21.594796 5690 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9af03e3a9c6487892664720ffec2a7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42143 } }
I20260927 02:44:21.595036 5690 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:21.595104 5690 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9af03e3a9c6487892664720ffec2a7c, State: Initialized, Role: FOLLOWER
I20260927 02:44:21.595484 5690 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9af03e3a9c6487892664720ffec2a7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42143 } }
I20260927 02:44:21.595630 5690 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:21.595700 5690 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:21.595788 5690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:21.596868 5690 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9af03e3a9c6487892664720ffec2a7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42143 } }
I20260927 02:44:21.597201 5690 leader_election.cc:302] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9af03e3a9c6487892664720ffec2a7c; no voters:
I20260927 02:44:21.597474 5690 leader_election.cc:288] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:21.597620 5695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:21.597847 5695 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [term 1 LEADER]: Becoming Leader. State: Replica: b9af03e3a9c6487892664720ffec2a7c, State: Running, Role: LEADER
I20260927 02:44:21.598124 5695 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9af03e3a9c6487892664720ffec2a7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42143 } }
I20260927 02:44:21.598647 5690 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:21.599779 5697 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9af03e3a9c6487892664720ffec2a7c. Latest consensus state: current_term: 1 leader_uuid: "b9af03e3a9c6487892664720ffec2a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9af03e3a9c6487892664720ffec2a7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42143 } } }
I20260927 02:44:21.599779 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9af03e3a9c6487892664720ffec2a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9af03e3a9c6487892664720ffec2a7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42143 } } }
I20260927 02:44:21.600085 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:21.600111 5697 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:21.600963 5705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:21.603330 5705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:21.608237 5705 catalog_manager.cc:1347] Generated new cluster ID: d54c87ba1d2e45e2b122146c7dd61b7c
I20260927 02:44:21.608294 5705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:21.628311 5705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:21.629475 5705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:21.638226 5705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9af03e3a9c6487892664720ffec2a7c: Generated new TSK 0
I20260927 02:44:21.638978 5705 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:44:21.677395 5720 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 02:44:21.677475 5720 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:44:21.678074 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.654267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.0.98.62:42143)
0927 02:44:21.654404 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:21.654408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:21.654618 (+ 210us) server_negotiation.cc:407] Connection header received
0927 02:44:21.654757 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:21.654759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:21.654803 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:21.654892 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:44:21.655149 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.655662 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.656954 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.657191 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.657276 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.657591 (+ 315us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:44:21.677703 (+ 20112us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:44:21.677972 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 02:44:21.678153 5693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 02:44:21.678196 5719 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:21.654193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42143 (local address 127.0.0.1:33858)
0927 02:44:21.654498 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:21.654513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:21.654625 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:21.654912 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:21.654915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:21.654923 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:44:21.655044 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:21.655050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.655766 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.655770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:21.656784 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:21.656794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.657289 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.657307 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:21.657395 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.657421 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:44:21.677867 (+ 20446us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:44:21.678029 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42143: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":122,"threads_started":1}
I20260927 02:44:21.678965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 5623
2026-09-27T02:44:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (321 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 02:44:21.717358 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:44:21.718206 392 webserver.cc:460] Webserver started at https://127.0.0.1:39739/ using document root <none> and password file <none>
I20260927 02:44:21.718334 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:44:21.718606 392 webserver.cc:460] Webserver started at http://127.0.0.1:42441/ using document root <none> and password file <none>
2026-09-27T02:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:21Z Disabled control of system clock
I20260927 02:44:21.726948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40089
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40089
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39739/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:44:21.831833 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:21.832079 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:21.832150 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:21.836077 5726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:21.836127 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:21.836143 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:21.836159 5726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:21.836217 5726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:44:21.836247 5726 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:44:21.836275 5726 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39739/jwks.json
I20260927 02:44:21.840888 5726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39739/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:21.841863 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:21.843146 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:21.851483 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:21.851605 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:21.851697 5726 server_base.cc:1034] running on GCE node
W20260927 02:44:21.851802 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:21.852409 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:21.853430 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:21.854619 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790477061854585 us; error 44 us; skew 500 ppm
I20260927 02:44:21.856520 5726 webserver.cc:460] Webserver started at http://127.0.98.62:34989/ using document root <none> and password file <none>
I20260927 02:44:21.857069 5726 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:21.857129 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:21.857359 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:21.859278 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "15800cc191cb4e72873df7b088b00a68"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "0f6d057bd7e0d65bf9490b54b99bac0d"
server_key_iv: "ee6f8d6d44ae0b14ca0879f17567457f"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.859815 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "15800cc191cb4e72873df7b088b00a68"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "0f6d057bd7e0d65bf9490b54b99bac0d"
server_key_iv: "ee6f8d6d44ae0b14ca0879f17567457f"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.863304 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:21.865649 5740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:21.866775 5726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:44:21.866909 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "15800cc191cb4e72873df7b088b00a68"
format_stamp: "Formatted at 2026-09-27 02:44:21 on dist-test-slave-gkrl"
server_key: "0f6d057bd7e0d65bf9490b54b99bac0d"
server_key_iv: "ee6f8d6d44ae0b14ca0879f17567457f"
server_key_version: "encryptionkey@0"
I20260927 02:44:21.867053 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:21.876967 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:21.877600 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:21.877780 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:21.884197 5726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40089
I20260927 02:44:21.884249 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40089 every 8 connection(s)
I20260927 02:44:21.885098 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:21.888239 5793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:21.891692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 5726
I20260927 02:44:21.891830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:21.892128 392 external_mini_cluster.cc:1428] Setting key 25472f51fdcafc71d363217e93b18627
I20260927 02:44:21.893777 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68: Bootstrap starting.
I20260927 02:44:21.895911 5793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:21.896649 5793 log.cc:824] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:21.898334 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68: No bootstrap required, opened a new log
I20260927 02:44:21.901012 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.893391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.0.98.62:40089)
0927 02:44:21.894068 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:21.894084 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:21.894120 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:44:21.894737 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:21.894752 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:21.895127 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:21.895445 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:21.895921 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.896974 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.897841 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.898291 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.898459 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.899305 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:21.899310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:21.899321 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:21.899693 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:21.899744 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:21.899762 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:21.899870 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:21.900132 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 02:44:21.900367 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260927 02:44:21.902082 5793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15800cc191cb4e72873df7b088b00a68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40089 } }
I20260927 02:44:21.902336 5793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:21.902380 5793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15800cc191cb4e72873df7b088b00a68, State: Initialized, Role: FOLLOWER
I20260927 02:44:21.902827 5793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15800cc191cb4e72873df7b088b00a68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40089 } }
I20260927 02:44:21.902977 5793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:21.903069 5793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:21.903168 5793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:21.903733 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:44:21.904189 5793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15800cc191cb4e72873df7b088b00a68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40089 } }
I20260927 02:44:21.904537 5793 leader_election.cc:302] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15800cc191cb4e72873df7b088b00a68; no voters:
I20260927 02:44:21.904891 5793 leader_election.cc:288] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:21.904955 5798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:21.905133 5798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [term 1 LEADER]: Becoming Leader. State: Replica: 15800cc191cb4e72873df7b088b00a68, State: Running, Role: LEADER
I20260927 02:44:21.905475 5798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15800cc191cb4e72873df7b088b00a68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40089 } }
I20260927 02:44:21.906056 5793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:21.907153 5800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15800cc191cb4e72873df7b088b00a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15800cc191cb4e72873df7b088b00a68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40089 } } }
I20260927 02:44:21.907189 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15800cc191cb4e72873df7b088b00a68. Latest consensus state: current_term: 1 leader_uuid: "15800cc191cb4e72873df7b088b00a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15800cc191cb4e72873df7b088b00a68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40089 } } }
I20260927 02:44:21.907263 5800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:21.907289 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:21.907632 5807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:21.910768 5807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:21.916038 5807 catalog_manager.cc:1347] Generated new cluster ID: 02a57da059c041849130c26ae476d04f
I20260927 02:44:21.916118 5807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:21.947610 5807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:21.948850 5807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:21.955965 5807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15800cc191cb4e72873df7b088b00a68: Generated new TSK 0
I20260927 02:44:21.956736 5807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:21.979411 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:21.974747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.0.98.62:40089)
0927 02:44:21.974883 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:21.974890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:21.975096 (+ 206us) server_negotiation.cc:407] Connection header received
0927 02:44:21.975284 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:21.975286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:21.975331 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:21.975443 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:44:21.975681 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.976321 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.977234 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.977460 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.977571 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.977898 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:44:21.979032 (+ 1134us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:44:21.979264 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 02:44:21.979532 5822 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:21.974667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40089 (local address 127.0.0.1:54992)
0927 02:44:21.975010 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:21.975021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:21.975144 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:21.975451 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:21.975454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:21.975462 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:44:21.975575 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:21.975581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.976479 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.976483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:21.977116 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:21.977123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:21.977570 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:21.977591 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:21.977683 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:21.977704 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:44:21.979201 (+ 1497us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:44:21.979333 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40089: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
W20260927 02:44:21.979787 5796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39739/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 02:44:21.980109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 5726
E20260927 02:44:21.992151 5823 webserver.cc:533] Webserver: sslize failed:
E20260927 02:44:21.992219 5823 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T02:44:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T02:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:22Z Disabled control of system clock
I20260927 02:44:22.027747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45161
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45161
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/testchainca.cert with env {}
W20260927 02:44:22.135169 5827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:22.135407 5827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:22.135445 5827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:22.139447 5827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:22.139503 5827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:22.139518 5827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:22.139535 5827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:22.139550 5827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:44:22.139595 5827 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:44:22.139624 5827 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 02:44:22.144984 5827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:22.145925 5827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:22.146859 5827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:22.155943 5832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:22.155937 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:22.156011 5827 server_base.cc:1034] running on GCE node
W20260927 02:44:22.155961 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:22.156687 5827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:22.157704 5827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:22.158865 5827 hybrid_clock.cc:648] HybridClock initialized: now 1790477062158854 us; error 27 us; skew 500 ppm
I20260927 02:44:22.160706 5827 webserver.cc:460] Webserver started at http://127.0.98.62:43447/ using document root <none> and password file <none>
I20260927 02:44:22.161254 5827 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:22.161315 5827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:22.161563 5827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:22.163472 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "5c324762b78a4797a31789f54ba8aaed"
format_stamp: "Formatted at 2026-09-27 02:44:22 on dist-test-slave-gkrl"
server_key: "efec9a12d7701d611bbe51a295e44d69"
server_key_iv: "06905d30f956826df0c6b6cfbcdb1224"
server_key_version: "encryptionkey@0"
I20260927 02:44:22.164067 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c324762b78a4797a31789f54ba8aaed"
format_stamp: "Formatted at 2026-09-27 02:44:22 on dist-test-slave-gkrl"
server_key: "efec9a12d7701d611bbe51a295e44d69"
server_key_iv: "06905d30f956826df0c6b6cfbcdb1224"
server_key_version: "encryptionkey@0"
I20260927 02:44:22.167464 5827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:22.169864 5841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:22.171006 5827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:22.171149 5827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "5c324762b78a4797a31789f54ba8aaed"
format_stamp: "Formatted at 2026-09-27 02:44:22 on dist-test-slave-gkrl"
server_key: "efec9a12d7701d611bbe51a295e44d69"
server_key_iv: "06905d30f956826df0c6b6cfbcdb1224"
server_key_version: "encryptionkey@0"
I20260927 02:44:22.171257 5827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:22.189966 5827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:22.190588 5827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:22.190778 5827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:22.197216 5827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45161
I20260927 02:44:22.197248 5893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45161 every 8 connection(s)
I20260927 02:44:22.198067 5827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:22.201017 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:22.203033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 5827
I20260927 02:44:22.203173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:22.203457 392 external_mini_cluster.cc:1428] Setting key c5c6b038fd5a374b31947b88bfce6743
I20260927 02:44:22.206774 5894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed: Bootstrap starting.
I20260927 02:44:22.208962 5894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:22.209668 5894 log.cc:824] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:22.211510 5894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed: No bootstrap required, opened a new log
I20260927 02:44:22.212244 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:22.204795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.0.98.62:45161)
0927 02:44:22.205378 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:22.205392 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:22.205424 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:44:22.206015 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:22.206030 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:22.206379 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:22.206704 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:22.207098 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:22.208049 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:22.208907 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:22.209417 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:22.209597 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:22.210493 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:22.210498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:22.210511 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:22.210859 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:22.210910 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:22.210928 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:22.211049 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:22.211217 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 02:44:22.211486 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
I20260927 02:44:22.214979 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:44:22.215485 5894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c324762b78a4797a31789f54ba8aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } }
I20260927 02:44:22.215713 5894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:22.215771 5894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c324762b78a4797a31789f54ba8aaed, State: Initialized, Role: FOLLOWER
I20260927 02:44:22.216223 5894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c324762b78a4797a31789f54ba8aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } }
I20260927 02:44:22.216377 5894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:22.216480 5894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:22.216595 5894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:22.217408 5894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c324762b78a4797a31789f54ba8aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } }
I20260927 02:44:22.217734 5894 leader_election.cc:302] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c324762b78a4797a31789f54ba8aaed; no voters:
I20260927 02:44:22.218015 5894 leader_election.cc:288] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:22.218137 5899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:22.218356 5899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [term 1 LEADER]: Becoming Leader. State: Replica: 5c324762b78a4797a31789f54ba8aaed, State: Running, Role: LEADER
I20260927 02:44:22.218721 5899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c324762b78a4797a31789f54ba8aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } }
I20260927 02:44:22.219151 5894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:22.220404 5900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c324762b78a4797a31789f54ba8aaed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c324762b78a4797a31789f54ba8aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } } }
I20260927 02:44:22.220448 5901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c324762b78a4797a31789f54ba8aaed. Latest consensus state: current_term: 1 leader_uuid: "5c324762b78a4797a31789f54ba8aaed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c324762b78a4797a31789f54ba8aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } } }
I20260927 02:44:22.220526 5900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:22.220538 5901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:22.221567 5908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:22.224117 5908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:22.228866 5908 catalog_manager.cc:1347] Generated new cluster ID: 8496528efa7a488389559b5312d3c191
I20260927 02:44:22.228927 5908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:22.235294 5908 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:22.236143 5908 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:22.251122 5908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c324762b78a4797a31789f54ba8aaed: Generated new TSK 0
I20260927 02:44:22.251724 5908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:22.712267 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:22.264405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.98.62:45161)
0927 02:44:22.264529 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:22.264541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:22.264699 (+ 158us) server_negotiation.cc:407] Connection header received
0927 02:44:22.264815 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:22.264818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:22.264872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:22.264984 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:44:22.265189 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:22.266007 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:22.266836 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:22.267178 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:22.267277 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:22.267551 (+ 274us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:44:22.711773 (+444222us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:44:22.712148 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60130: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 02:44:22.712338 5897 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60130: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 02:44:22.712459 5923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:22.264319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45161 (local address 127.0.0.1:60130)
0927 02:44:22.264618 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:22.264629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:22.264718 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:22.264993 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:22.264995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:22.265001 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:44:22.265090 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:22.265095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:22.266117 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:22.266119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:22.266736 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:22.266742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:22.267278 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:22.267296 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:22.267365 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:22.267386 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:44:22.712083 (+444697us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:44:22.712255 (+ 172us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45161: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":109,"threads_started":1}
I20260927 02:44:22.713128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 5827
2026-09-27T02:44:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (712 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (116 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (116 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:22 dist-test-slave-gkrl krb5kdc[5935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:22 dist-test-slave-gkrl krb5kdc[5935](info): set up 2 sockets
Sep 27 02:44:22 dist-test-slave-gkrl krb5kdc[5935](info): commencing operation
krb5kdc: starting...
W20260927 02:44:23.980232 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:24.098719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:24.205173 5948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:24.205415 5948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:24.205453 5948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:24.209349 5948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:24.209395 5948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:24.209414 5948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:24.209430 5948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:24.209444 5948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:24.214576 5948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:24.215719 5948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:24.216688 5948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:24.224743 5954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:24.224743 5953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:24.224761 5956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:24.225097 5948 server_base.cc:1034] running on GCE node
I20260927 02:44:24.225621 5948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:24.226686 5948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:24.227916 5948 hybrid_clock.cc:648] HybridClock initialized: now 1790477064227859 us; error 54 us; skew 500 ppm
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:24.230651 5948 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:24.231478 5948 webserver.cc:460] Webserver started at http://127.0.98.62:42027/ using document root <none> and password file <none>
I20260927 02:44:24.232019 5948 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:24.232084 5948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:24.232257 5948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:24.234164 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "c3f81d791b3d468abc6694f381713697"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "99d56955a11940cbef339da4e302e55f"
server_key_iv: "cfc2bb771b812c1ba76abdcf6573b95a"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.234640 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3f81d791b3d468abc6694f381713697"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "99d56955a11940cbef339da4e302e55f"
server_key_iv: "cfc2bb771b812c1ba76abdcf6573b95a"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.238019 5948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:24.240164 5963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.241314 5948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:44:24.241425 5948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "c3f81d791b3d468abc6694f381713697"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "99d56955a11940cbef339da4e302e55f"
server_key_iv: "cfc2bb771b812c1ba76abdcf6573b95a"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.241504 5948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:24.254808 5948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:24.257802 5948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:24.257936 5948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:24.264858 5948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33301
I20260927 02:44:24.265130 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33301 every 8 connection(s)
I20260927 02:44:24.265836 5948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:24.268479 6016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:24.273459 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697: Bootstrap starting.
I20260927 02:44:24.274164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 5948
I20260927 02:44:24.274293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:24.274585 392 external_mini_cluster.cc:1428] Setting key b3ff437f8b336ae1c519b78ec928cf75
I20260927 02:44:24.276130 6016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:24.276949 6016 log.cc:824] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:24.278945 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697: No bootstrap required, opened a new log
I20260927 02:44:24.282702 6016 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f81d791b3d468abc6694f381713697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } }
I20260927 02:44:24.282894 6016 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:24.282934 6016 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f81d791b3d468abc6694f381713697, State: Initialized, Role: FOLLOWER
I20260927 02:44:24.283381 6016 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f81d791b3d468abc6694f381713697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } }
I20260927 02:44:24.283528 6016 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:24.283615 6016 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:24.283732 6016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:24.284705 6016 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f81d791b3d468abc6694f381713697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } }
I20260927 02:44:24.285069 6016 leader_election.cc:302] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f81d791b3d468abc6694f381713697; no voters:
I20260927 02:44:24.285408 6016 leader_election.cc:288] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:24.285519 6021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:24.285702 6021 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [term 1 LEADER]: Becoming Leader. State: Replica: c3f81d791b3d468abc6694f381713697, State: Running, Role: LEADER
I20260927 02:44:24.285979 6021 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f81d791b3d468abc6694f381713697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } }
I20260927 02:44:24.286667 6016 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:24.286988 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f81d791b3d468abc6694f381713697. Latest consensus state: current_term: 1 leader_uuid: "c3f81d791b3d468abc6694f381713697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f81d791b3d468abc6694f381713697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } }
I20260927 02:44:24.286998 6022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f81d791b3d468abc6694f381713697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f81d791b3d468abc6694f381713697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } }
I20260927 02:44:24.287189 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:24.287485 6022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:24.287822 6025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:24.290143 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:24.275664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.0.98.62:33301)
0927 02:44:24.276157 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:24.276176 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:24.276218 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:44:24.277019 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:24.277041 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:24.277417 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:24.277739 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:24.278406 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.279825 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.280704 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.281410 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.281610 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:24.284640 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:24.284649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:24.284666 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:24.284697 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:24.286651 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:24.287172 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:24.287178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:24.287184 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:24.287428 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:24.287792 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:24.287796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:24.287797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:24.288301 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:24.288485 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:24.288936 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 02:44:24.289230 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":167,"threads_started":1}
I20260927 02:44:24.291080 6025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:24.296702 6025 catalog_manager.cc:1347] Generated new cluster ID: 0a7c76fe78594cf08c0a7737d4400819
I20260927 02:44:24.296759 6025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:24.305056 6025 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:24.305878 6025 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:24.316128 6025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f81d791b3d468abc6694f381713697: Generated new TSK 0
I20260927 02:44:24.316823 6025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:24.348608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33301
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:24.454684 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:24.454905 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:24.454945 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:24.458874 6042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:24.458923 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:24.459014 6042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:24.464152 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:24.465109 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:24.466073 6042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:24.475387 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:24.475382 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:24.475449 6042 server_base.cc:1034] running on GCE node
W20260927 02:44:24.475396 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:24.476136 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:24.476814 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:24.478013 6042 hybrid_clock.cc:648] HybridClock initialized: now 1790477064478005 us; error 48 us; skew 500 ppm
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:24.480943 6042 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:24.481772 6042 webserver.cc:460] Webserver started at http://127.0.98.1:46511/ using document root <none> and password file <none>
I20260927 02:44:24.482327 6042 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:24.482436 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:24.482669 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:24.484761 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "40fe2f346a4a443c86266c36c93ad386"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "fd752cc6f79612fdac3870310e7ddca0"
server_key_iv: "14c7d26ede570bae9812643861de5d10"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.485330 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40fe2f346a4a443c86266c36c93ad386"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "fd752cc6f79612fdac3870310e7ddca0"
server_key_iv: "14c7d26ede570bae9812643861de5d10"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.489148 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 02:44:24.491496 6057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.492760 6042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:44:24.492904 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "40fe2f346a4a443c86266c36c93ad386"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "fd752cc6f79612fdac3870310e7ddca0"
server_key_iv: "14c7d26ede570bae9812643861de5d10"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.493014 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:24.510771 6042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:24.513680 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:24.513877 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:24.514444 6042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:24.515259 6042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:24.515323 6042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.515380 6042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:24.515419 6042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.524580 6042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40843
I20260927 02:44:24.524601 6170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40843 every 8 connection(s)
I20260927 02:44:24.525460 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:24.534358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 6042
I20260927 02:44:24.534516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:24.534782 392 external_mini_cluster.cc:1428] Setting key d75f06ecddbc38d786125a1b2457f68a
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:24.539239 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:24.527281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51365 (local address 127.0.98.62:33301)
0927 02:44:24.527426 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:24.527430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:24.528270 (+ 840us) server_negotiation.cc:407] Connection header received
0927 02:44:24.529661 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:24.529664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:24.529720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:24.529822 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:24.530917 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.531485 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.532670 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.532911 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.533024 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:24.535568 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:24.535572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:24.535574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:24.535597 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:24.537259 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:24.537766 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:24.537771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:24.537772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:24.537855 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:24.538225 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:24.538228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:24.538230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:24.538400 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:24.538515 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:24.538956 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 02:44:24.539068 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:44:24.540061 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:24.527637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.98.1:51365)
0927 02:44:24.528132 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:24.528172 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:24.529450 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:24.529988 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:24.529994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:24.530330 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:24.530770 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:24.530782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.531613 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.531619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:24.532544 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:24.532553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.533055 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.533059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:24.533224 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:24.533778 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:24.533794 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:24.535430 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:24.537394 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:24.537399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:24.537414 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:24.537658 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:24.537964 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:24.537967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:24.537969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:24.538112 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:24.538512 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:24.538517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:24.538732 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 02:44:24.538914 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 02:44:24.539204 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":121,"threads_started":1}
I20260927 02:44:24.541497 6171 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33301
I20260927 02:44:24.541733 6171 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:24.542188 6171 heartbeater.cc:511] Master 127.0.98.62:33301 requested a full tablet report, sending...
I20260927 02:44:24.543865 5980 ts_manager.cc:194] Registered new tserver with Master: 40fe2f346a4a443c86266c36c93ad386 (127.0.98.1:40843)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:44:24.545209 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:24.561177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33301
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:24.667380 6176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:24.667630 6176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:24.667671 6176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:24.671820 6176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:24.671921 6176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:24.672029 6176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:24.677105 6176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:24.678040 6176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:24.679003 6176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:24.688377 6181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:24.688436 6184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:24.688377 6182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:24.688611 6176 server_base.cc:1034] running on GCE node
I20260927 02:44:24.689109 6176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:24.689700 6176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:24.690909 6176 hybrid_clock.cc:648] HybridClock initialized: now 1790477064690894 us; error 25 us; skew 500 ppm
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:24.693559 6176 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:24.694367 6176 webserver.cc:460] Webserver started at http://127.0.98.2:38855/ using document root <none> and password file <none>
I20260927 02:44:24.694936 6176 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:24.695030 6176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:24.695248 6176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:24.697340 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad57c9fc7f774fd9868b448f50649352"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "d501d1a971842a32610bd032a500f4ec"
server_key_iv: "0073a99cd5157f8025310218c55b5c0c"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.697925 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad57c9fc7f774fd9868b448f50649352"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "d501d1a971842a32610bd032a500f4ec"
server_key_iv: "0073a99cd5157f8025310218c55b5c0c"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.701361 6176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:44:24.703579 6191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.705106 6176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:44:24.705212 6176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "ad57c9fc7f774fd9868b448f50649352"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "d501d1a971842a32610bd032a500f4ec"
server_key_iv: "0073a99cd5157f8025310218c55b5c0c"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.705323 6176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:24.734284 6176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:24.737659 6176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:24.737808 6176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:24.738322 6176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:24.739090 6176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:24.739137 6176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.739176 6176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:24.739192 6176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.748208 6176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41101
I20260927 02:44:24.748284 6304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41101 every 8 connection(s)
I20260927 02:44:24.749107 6176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:24.756927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 6176
I20260927 02:44:24.757064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:24.757441 392 external_mini_cluster.cc:1428] Setting key ff2bfb835bae00184b21fa188f2adec6
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:24.762882 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:24.750907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35813 (local address 127.0.98.62:33301)
0927 02:44:24.751033 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:24.751037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:24.751849 (+ 812us) server_negotiation.cc:407] Connection header received
0927 02:44:24.753234 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:24.753238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:24.753296 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:24.753375 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:24.754490 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.755020 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.756205 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.756436 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.756547 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:24.759129 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:24.759132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:24.759135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:24.759161 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:24.760802 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:24.761428 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:24.761431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:24.761432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:24.761500 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:24.761877 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:24.761879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:24.761880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:24.762031 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:24.762105 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:24.762685 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 02:44:24.762771 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:44:24.764192 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:24.751279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.98.2:35813)
0927 02:44:24.751703 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:24.751744 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:24.753019 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:24.753560 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:24.753567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:24.753895 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:24.754344 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:24.754356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.755152 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.755158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:24.756075 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:24.756084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.756587 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.756591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:24.756748 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:24.757381 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:24.757401 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:24.758972 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:24.760915 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:24.760921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:24.760978 (+ 57us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:24.761331 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:24.761589 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:24.761593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:24.761595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:24.761786 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:24.762135 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:24.762144 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:24.762449 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 02:44:24.762653 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 02:44:24.763147 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":118,"threads_started":1}
I20260927 02:44:24.765379 6305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33301
I20260927 02:44:24.765604 6305 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:24.766073 6305 heartbeater.cc:511] Master 127.0.98.62:33301 requested a full tablet report, sending...
I20260927 02:44:24.767087 5980 ts_manager.cc:194] Registered new tserver with Master: ad57c9fc7f774fd9868b448f50649352 (127.0.98.2:41101)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:24.767685 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:24.786906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33301
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:24.892658 6310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:24.892927 6310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:24.893008 6310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:24.896996 6310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:24.897071 6310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:24.897176 6310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:24.902230 6310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:24.903188 6310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:24.904192 6310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:24.913167 6316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:24.913168 6318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:24.913179 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:24.913341 6310 server_base.cc:1034] running on GCE node
I20260927 02:44:24.913879 6310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:24.914422 6310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:24.915607 6310 hybrid_clock.cc:648] HybridClock initialized: now 1790477064915592 us; error 25 us; skew 500 ppm
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:24.918452 6310 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:24.919307 6310 webserver.cc:460] Webserver started at http://127.0.98.3:44679/ using document root <none> and password file <none>
I20260927 02:44:24.919834 6310 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:24.919961 6310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:24.920185 6310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:24.922087 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "86caac3d265e41c0a9328e16a32b7f71"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "c9160d822c3fec3f60eea0b6c00d49b5"
server_key_iv: "55bbc850c3e439ecf0e7426f7aaacbb9"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.922631 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "86caac3d265e41c0a9328e16a32b7f71"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "c9160d822c3fec3f60eea0b6c00d49b5"
server_key_iv: "55bbc850c3e439ecf0e7426f7aaacbb9"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.926105 6310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:44:24.928507 6325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.929647 6310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:24.929780 6310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "86caac3d265e41c0a9328e16a32b7f71"
format_stamp: "Formatted at 2026-09-27 02:44:24 on dist-test-slave-gkrl"
server_key: "c9160d822c3fec3f60eea0b6c00d49b5"
server_key_iv: "55bbc850c3e439ecf0e7426f7aaacbb9"
server_key_version: "encryptionkey@0"
I20260927 02:44:24.929883 6310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:24.950901 6310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:24.954177 6310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:24.954375 6310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:24.954939 6310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:24.955705 6310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:24.955775 6310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.955847 6310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:24.955924 6310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:24.964767 6310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32795
I20260927 02:44:24.964900 6438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32795 every 8 connection(s)
I20260927 02:44:24.965672 6310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:24.972736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 6310
I20260927 02:44:24.972931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:24.973212 392 external_mini_cluster.cc:1428] Setting key e33c27a80615c6154ac48a9cea27639f
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:24.978993 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:24.967406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56139 (local address 127.0.98.62:33301)
0927 02:44:24.967565 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:24.967569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:24.968352 (+ 783us) server_negotiation.cc:407] Connection header received
0927 02:44:24.969717 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:24.969721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:24.969777 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:24.969891 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:24.970985 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.971504 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.972675 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.972944 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.973056 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:24.975754 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:24.975757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:24.975759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:24.975784 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:24.977244 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:24.977696 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:24.977699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:24.977700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:24.977771 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:24.978084 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:24.978086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:24.978087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:24.978251 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:24.978323 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:24.978783 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 02:44:24.978883 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 02:44:24.979737 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:24.967748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.98.3:56139)
0927 02:44:24.968190 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:24.968230 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:24.969506 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:24.970052 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:24.970059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:24.970398 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:24.970833 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:24.970846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.971626 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.971631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:24.972537 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:24.972547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:24.973087 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:24.973091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:24.973256 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:24.974022 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:24.974064 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:24.975612 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:24.977346 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:24.977350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:24.977361 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:24.977607 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:24.977849 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:24.977851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:24.977853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:24.978003 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:24.978335 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:24.978340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:24.978543 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 02:44:24.978690 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 02:44:24.978911 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":132,"threads_started":1}
I20260927 02:44:24.980885 6439 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33301
I20260927 02:44:24.981149 6439 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:24.981603 6439 heartbeater.cc:511] Master 127.0.98.62:33301 requested a full tablet report, sending...
I20260927 02:44:24.982604 5980 ts_manager.cc:194] Registered new tserver with Master: 86caac3d265e41c0a9328e16a32b7f71 (127.0.98.3:32795)
I20260927 02:44:24.983152 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56139
I20260927 02:44:24.987069 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/test-admin.keytab.
Sep 27 02:44:24 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:25.002454 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 02:44:25.005201 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:25.003714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.0.98.1:40843)
0927 02:44:25.004070 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:25.004083 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:25.004216 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:44:25.004308 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:25.004313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:25.004481 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:25.004595 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:25.005045 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41300: BlockingRecv error: recv got EOF from 127.0.0.1:41300 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":86,"threads_started":1}
W20260927 02:44:25.005594 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5cc)
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:25.006570 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:25.007484 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:44:25.017443 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:25.008388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.0.98.1:40843)
0927 02:44:25.008511 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:25.008514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:25.008530 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:25.008639 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:25.008641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:25.008678 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:25.008744 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:25.009346 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:25.010097 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:25.011088 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:25.011442 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:25.011537 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:25.013837 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:25.013847 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:25.013864 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:25.013891 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:25.015460 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:25.015848 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:25.015853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:25.015854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:25.016077 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:25.016472 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:25.016475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:25.016498 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:25.016728 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:25.016980 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:25.017147 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 02:44:25.017309 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 02:44:25.021230 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:25.020184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.98.1:40843)
0927 02:44:25.020313 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:25.020317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:25.020424 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:44:25.020576 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:25.020579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:25.020616 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:25.020703 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:25.021119 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41320: BlockingRecv error: recv got EOF from 127.0.0.1:41320 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 02:44:25.021458 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:25.022528 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:25.023393 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:44:25.032007 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:25.024226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.0.98.1:40843)
0927 02:44:25.024380 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:25.024384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:25.024478 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:44:25.024609 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:25.024612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:25.024647 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:25.024735 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:25.025254 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:25.025761 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:25.026521 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:25.026757 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:25.026865 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:25.029104 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:25.029108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:25.029110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:25.029139 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:25.030390 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:25.030818 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:25.030822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:25.030824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:25.030902 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:25.031178 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:25.031182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:25.031184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:25.031340 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:25.031470 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:25.031715 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 02:44:25.031820 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 02:44:25.034682 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:25.033757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.0.98.1:40843)
0927 02:44:25.033862 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:25.033866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:25.033910 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:44:25.034048 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:25.034050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:25.034084 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:25.034153 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:25.034584 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41338: BlockingRecv error: recv got EOF from 127.0.0.1:41338 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 02:44:25.034842 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477031916752-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:25.035751 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:25.036715 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:44:25.044642 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:25.037416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.0.98.1:40843)
0927 02:44:25.037550 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:25.037555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:25.037625 (+ 70us) server_negotiation.cc:407] Connection header received
0927 02:44:25.037711 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:25.037713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:25.037745 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:25.037800 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:25.038265 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:25.038728 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:25.039448 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:25.039643 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:25.039713 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:25.041799 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:25.041802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:25.041803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:25.041822 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:25.043024 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:25.043409 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:25.043412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:25.043413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:25.043486 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:25.043725 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:25.043728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:25.043729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:25.044004 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:25.044106 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:25.044356 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:44:25.044499 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 02:44:25.045575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 6042
I20260927 02:44:25.051716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 6176
I20260927 02:44:25.056969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 6310
I20260927 02:44:25.062019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 5948
2026-09-27T02:44:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2118 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[6459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[6459](info): set up 2 sockets
Sep 27 02:44:25 dist-test-slave-gkrl krb5kdc[6459](info): commencing operation
krb5kdc: starting...
W20260927 02:44:26.116302 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:26 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:44:26.234958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:26.341506 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:26.341787 6472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:26.341864 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:26.345911 6472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:26.346052 6472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:26.346122 6472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:26.346163 6472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:26.346199 6472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:26.351308 6472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:26.352370 6472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:26.353333 6472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:26.361778 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:26.361776 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:26.361789 6480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:26.361826 6472 server_base.cc:1034] running on GCE node
I20260927 02:44:26.362586 6472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:26.363685 6472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:26.364872 6472 hybrid_clock.cc:648] HybridClock initialized: now 1790477066364877 us; error 90 us; skew 500 ppm
Sep 27 02:44:26 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:26.371343 6472 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:44:26.372282 6472 webserver.cc:460] Webserver started at http://127.0.98.62:44873/ using document root <none> and password file <none>
I20260927 02:44:26.372782 6472 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:26.372839 6472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:26.373006 6472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:26.374877 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "ab504ffef5174fef83297fcd3dcf61db"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "a04096b978753390f0961c73060af7e2"
server_key_iv: "e2530040e50bc77d7a7183c4c4aa8bb3"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.375375 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab504ffef5174fef83297fcd3dcf61db"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "a04096b978753390f0961c73060af7e2"
server_key_iv: "e2530040e50bc77d7a7183c4c4aa8bb3"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.378882 6472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:44:26.381323 6487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:26.382609 6472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:44:26.382709 6472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "ab504ffef5174fef83297fcd3dcf61db"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "a04096b978753390f0961c73060af7e2"
server_key_iv: "e2530040e50bc77d7a7183c4c4aa8bb3"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.382787 6472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:26.395411 6472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:26.398197 6472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:26.398340 6472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:26.404943 6472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42067
I20260927 02:44:26.405225 6539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42067 every 8 connection(s)
I20260927 02:44:26.405861 6472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:26.408670 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:26.410214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 6472
I20260927 02:44:26.410353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:26.410635 392 external_mini_cluster.cc:1428] Setting key 8a6abc93525f19badabc36592c20ddc8
I20260927 02:44:26.414479 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: Bootstrap starting.
I20260927 02:44:26.416644 6540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:26.417541 6540 log.cc:824] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:44:26 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:26.419628 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: No bootstrap required, opened a new log
I20260927 02:44:26.424331 6540 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } }
I20260927 02:44:26.424576 6540 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:26.424631 6540 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab504ffef5174fef83297fcd3dcf61db, State: Initialized, Role: FOLLOWER
I20260927 02:44:26.424727 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:26.412001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.0.98.62:42067)
0927 02:44:26.412402 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:26.412417 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:26.412462 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:44:26.413061 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:26.413078 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:26.413382 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:26.413716 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:26.414495 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.415467 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.416359 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.416891 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.417044 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:26.419727 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:26.419734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:26.419750 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:26.419780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:26.421814 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:26.422235 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:26.422243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:26.422250 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:26.422408 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:26.422650 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:26.422653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:26.422655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:26.423128 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:26.423328 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:26.423502 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 02:44:26.423786 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
I20260927 02:44:26.425091 6540 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } }
I20260927 02:44:26.425243 6540 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:26.425307 6540 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:26.425393 6540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:26.426283 6540 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } }
I20260927 02:44:26.426644 6540 leader_election.cc:302] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab504ffef5174fef83297fcd3dcf61db; no voters:
I20260927 02:44:26.426973 6540 leader_election.cc:288] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:26.427081 6545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:26.427304 6545 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 1 LEADER]: Becoming Leader. State: Replica: ab504ffef5174fef83297fcd3dcf61db, State: Running, Role: LEADER
I20260927 02:44:26.427592 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } }
I20260927 02:44:26.428164 6540 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:26.429342 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab504ffef5174fef83297fcd3dcf61db. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } }
I20260927 02:44:26.429471 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:26.429596 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } }
I20260927 02:44:26.429708 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:26.429869 6554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:26.432556 6554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:26.437546 6554 catalog_manager.cc:1347] Generated new cluster ID: e59f773e4b784f129d6a5bc439c8053a
I20260927 02:44:26.437608 6554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:26.482322 6554 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:26.483673 6554 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:26.499703 6554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: Generated new TSK 0
I20260927 02:44:26.500581 6554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:44:26.542881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:26.653352 6566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:26.653618 6566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:26.653692 6566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:26.657579 6566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:26.657657 6566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:26.657745 6566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:26.662740 6566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:26.663702 6566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:26.664681 6566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:26.673694 6574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:26.673687 6571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:26.673866 6572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:26.674031 6566 server_base.cc:1034] running on GCE node
I20260927 02:44:26.674432 6566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:26.675015 6566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:26.676225 6566 hybrid_clock.cc:648] HybridClock initialized: now 1790477066676199 us; error 32 us; skew 500 ppm
Sep 27 02:44:26 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:26.679140 6566 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:44:26.680076 6566 webserver.cc:460] Webserver started at http://127.0.98.1:39669/ using document root <none> and password file <none>
I20260927 02:44:26.680608 6566 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:26.680703 6566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:26.680941 6566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:26.682883 6566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "9d39bf310b3a486bae2e100f6115f8f9"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "4f8e4158c932370c546793adb27f5b5c"
server_key_iv: "15605ddab1d6ceb528e0c52f68bd670d"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.683427 6566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9d39bf310b3a486bae2e100f6115f8f9"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "4f8e4158c932370c546793adb27f5b5c"
server_key_iv: "15605ddab1d6ceb528e0c52f68bd670d"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.687156 6566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 02:44:26.689479 6581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:26.690696 6566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:26.690821 6566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "9d39bf310b3a486bae2e100f6115f8f9"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "4f8e4158c932370c546793adb27f5b5c"
server_key_iv: "15605ddab1d6ceb528e0c52f68bd670d"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.690955 6566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:26.737149 6566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:26.740319 6566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:26.740530 6566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:26.741114 6566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:26.741923 6566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:26.741993 6566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:26.742065 6566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:26.742108 6566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:26.751258 6566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33107
I20260927 02:44:26.751286 6694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33107 every 8 connection(s)
I20260927 02:44:26.752195 6566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:26.759719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 6566
I20260927 02:44:26.759868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:26.760207 392 external_mini_cluster.cc:1428] Setting key 65a46b72e3181d267e4db98798557176
Sep 27 02:44:26 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:26.765978 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:26.753947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60145 (local address 127.0.98.62:42067)
0927 02:44:26.754085 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:26.754089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:26.754887 (+ 798us) server_negotiation.cc:407] Connection header received
0927 02:44:26.756218 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:26.756221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:26.756266 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:26.756350 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:26.757439 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.757979 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.759030 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.759257 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.759349 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:26.762080 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:26.762083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:26.762085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:26.762108 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:26.763763 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:26.764303 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:26.764307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:26.764310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:26.764409 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:26.764732 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:26.764739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:26.764744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:26.764995 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:26.765116 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:26.765598 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 02:44:26.765742 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:44:26.766659 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:26.754306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:60145)
0927 02:44:26.754726 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:26.754768 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:26.756008 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:26.756493 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:26.756500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:26.756847 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:26.757307 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:26.757319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.758081 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.758084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:26.758921 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:26.758930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.759381 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.759384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:26.759550 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:26.760172 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:26.760187 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:26.761896 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:26.763928 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:26.763934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:26.763953 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:26.764203 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:26.764504 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:26.764506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:26.764508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:26.764630 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:26.765117 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:26.765122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:26.765334 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 02:44:26.765488 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 02:44:26.765792 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260927 02:44:26.767925 6695 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260927 02:44:26.768141 6695 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:26.768605 6695 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:44:26.770090 6504 ts_manager.cc:194] Registered new tserver with Master: 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107)
I20260927 02:44:26.771180 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:44:26.786343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:26.894205 6700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:26.894444 6700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:26.894485 6700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:26.898456 6700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:26.898514 6700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:26.898615 6700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:26.903828 6700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:26.904870 6700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:26.905864 6700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:26.915279 6705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:26.915262 6708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:26.915256 6706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:26.915407 6700 server_base.cc:1034] running on GCE node
I20260927 02:44:26.915970 6700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:26.916641 6700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:26.917798 6700 hybrid_clock.cc:648] HybridClock initialized: now 1790477066917781 us; error 30 us; skew 500 ppm
Sep 27 02:44:26 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:26.920447 6700 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:44:26.921422 6700 webserver.cc:460] Webserver started at http://127.0.98.2:40835/ using document root <none> and password file <none>
I20260927 02:44:26.921901 6700 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:26.921962 6700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:26.922113 6700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:26.924054 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "26e71a93b4414e3bb237b4bbf92be938"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "e6e72ce2abe6996b6f4451c29c3c2760"
server_key_iv: "ea78790e161a78d17eff0f56045cc1a5"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.924539 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "26e71a93b4414e3bb237b4bbf92be938"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "e6e72ce2abe6996b6f4451c29c3c2760"
server_key_iv: "ea78790e161a78d17eff0f56045cc1a5"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.928015 6700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:26.930367 6715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:26.931486 6700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:26.931628 6700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "26e71a93b4414e3bb237b4bbf92be938"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "e6e72ce2abe6996b6f4451c29c3c2760"
server_key_iv: "ea78790e161a78d17eff0f56045cc1a5"
server_key_version: "encryptionkey@0"
I20260927 02:44:26.931733 6700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:26.964040 6700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:26.967007 6700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:26.967212 6700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:26.967813 6700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:26.968657 6700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:26.968727 6700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:26.968799 6700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:26.968851 6700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:26.977434 6700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45807
I20260927 02:44:26.977515 6828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45807 every 8 connection(s)
I20260927 02:44:26.978339 6700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:26.982234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 6700
I20260927 02:44:26.982352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:26.982666 392 external_mini_cluster.cc:1428] Setting key cccd06c881ccb341456e7be8b6160d4a
Sep 27 02:44:26 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:26.992815 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:26.980145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50675 (local address 127.0.98.62:42067)
0927 02:44:26.980304 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:26.980309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:26.981150 (+ 841us) server_negotiation.cc:407] Connection header received
0927 02:44:26.982471 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:26.982474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:26.982535 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:26.982632 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:26.983976 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.984509 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.985724 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.985970 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.986088 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:26.989031 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:26.989034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:26.989036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:26.989058 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:26.990538 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:26.991155 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:26.991158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:26.991159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:26.991235 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:26.991668 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:26.991671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:26.991672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:26.991855 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:26.992036 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:26.992606 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 02:44:26.992703 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 02:44:26.993562 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:26.980567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:50675)
0927 02:44:26.980987 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:26.981043 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:26.982204 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:26.982821 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:26.982828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:26.983185 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:26.983772 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:26.983792 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.984662 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.984665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:26.985605 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:26.985614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:26.986134 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:26.986138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:26.986321 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:26.986922 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:26.986938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:26.988865 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:26.990652 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:26.990660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:26.990677 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:26.991026 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:26.991334 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:26.991337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:26.991341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:26.991544 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:26.992032 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:26.992041 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:26.992357 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 02:44:26.992525 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 02:44:26.992757 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 02:44:26.994957 6829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260927 02:44:26.995196 6829 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:26.995657 6829 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
I20260927 02:44:26.996773 6504 ts_manager.cc:194] Registered new tserver with Master: 26e71a93b4414e3bb237b4bbf92be938 (127.0.98.2:45807)
I20260927 02:44:26.997349 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:44:27.011482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:27.136030 6834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:27.136264 6834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:27.136339 6834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:27.140271 6834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:27.140323 6834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:27.140419 6834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:27.145439 6834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:27.146404 6834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:27.147353 6834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:27.156646 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.156703 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.156699 6839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:27.156929 6834 server_base.cc:1034] running on GCE node
I20260927 02:44:27.157403 6834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:27.158087 6834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:27.159286 6834 hybrid_clock.cc:648] HybridClock initialized: now 1790477067159260 us; error 48 us; skew 500 ppm
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:27.162213 6834 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:44:27.163147 6834 webserver.cc:460] Webserver started at http://127.0.98.3:45939/ using document root <none> and password file <none>
I20260927 02:44:27.163717 6834 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:27.163813 6834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:27.164091 6834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:27.166163 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "5162b030e0504b25a9388e542c12c2a8"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "6d3ced3c0a1135d0f723645dbb9d0d0b"
server_key_iv: "c4125df49c8a6397a0c10c9703d28b1d"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.166764 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5162b030e0504b25a9388e542c12c2a8"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "6d3ced3c0a1135d0f723645dbb9d0d0b"
server_key_iv: "c4125df49c8a6397a0c10c9703d28b1d"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.170540 6834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 02:44:27.172986 6849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.174209 6834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:27.174345 6834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "5162b030e0504b25a9388e542c12c2a8"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "6d3ced3c0a1135d0f723645dbb9d0d0b"
server_key_iv: "c4125df49c8a6397a0c10c9703d28b1d"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.174454 6834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:27.185112 6834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:27.188331 6834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:27.188542 6834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:27.189146 6834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:27.189993 6834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:27.190068 6834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.190145 6834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:27.190196 6834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.200217 6834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35271
I20260927 02:44:27.200256 6962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35271 every 8 connection(s)
I20260927 02:44:27.201153 6834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:27.207377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 6834
I20260927 02:44:27.207504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:27.207818 392 external_mini_cluster.cc:1428] Setting key 4716c716203b1ffadd094e7791b72721
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:27.215739 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.202923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46099 (local address 127.0.98.62:42067)
0927 02:44:27.203066 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.203070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.203951 (+ 881us) server_negotiation.cc:407] Connection header received
0927 02:44:27.205288 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.205291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.205351 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.205453 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.206973 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.207896 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.209269 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.209562 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.209649 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.212406 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.212409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.212412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.212437 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.213847 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.214356 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.214359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.214360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.214447 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.214757 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.214759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.214760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.214952 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.215029 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.215524 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.215625 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:44:27.216532 6965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.203322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:46099)
0927 02:44:27.203780 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.203821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.205072 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.205640 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.205647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.205996 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.206751 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.206767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.208025 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.208028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.209143 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.209157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.209709 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.209713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.209975 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.210621 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.210637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.212221 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.213965 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.213969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.213980 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.214247 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.214534 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.214536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.214538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.214671 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.215051 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.215055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.215266 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.215434 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.215672 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260927 02:44:27.217769 6963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260927 02:44:27.218050 6963 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:27.218585 6963 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
I20260927 02:44:27.219682 6504 ts_manager.cc:194] Registered new tserver with Master: 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271)
I20260927 02:44:27.220325 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46099
I20260927 02:44:27.221978 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 02:44:27.250554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39225
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42067,127.0.98.61:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:27.358502 6968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:27.358733 6968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:27.358770 6968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:27.362668 6968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:27.362717 6968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:27.362736 6968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:27.362753 6968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:27.362768 6968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:27.367800 6968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42067,127.0.98.61:39225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
W20260927 02:44:27.367973 6968 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42067,127.0.98.61:39225'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 02:44:27.368698 6968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:27.369566 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:27.377712 6973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.377707 6976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:27.377914 6968 server_base.cc:1034] running on GCE node
W20260927 02:44:27.377712 6974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:27.378532 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:27.379550 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:27.380709 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790477067380664 us; error 58 us; skew 500 ppm
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:27.383342 6968 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:44:27.384217 6968 webserver.cc:460] Webserver started at http://127.0.98.61:44203/ using document root <none> and password file <none>
I20260927 02:44:27.384745 6968 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:27.384833 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:27.385066 6968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:27.386999 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data/instance:
uuid: "f4b216448cab4b15b3e3f75ac5005c2e"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "c36af7e7246a891b986406c122edd1ea"
server_key_iv: "9efcc5953a749affa3818df5f39b4993"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.387562 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal/instance:
uuid: "f4b216448cab4b15b3e3f75ac5005c2e"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "c36af7e7246a891b986406c122edd1ea"
server_key_iv: "9efcc5953a749affa3818df5f39b4993"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.391089 6968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:44:27.393465 6983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.394973 6968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:27.395107 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
uuid: "f4b216448cab4b15b3e3f75ac5005c2e"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "c36af7e7246a891b986406c122edd1ea"
server_key_iv: "9efcc5953a749affa3818df5f39b4993"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.395228 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:27.406862 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:27.409703 6968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:27.409897 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:27.416878 6968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39225
I20260927 02:44:27.417140 7035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39225 every 8 connection(s)
I20260927 02:44:27.417874 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:44:27.420580 7036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:27.424167 7036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:27.426265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 6968
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:44:27.437934 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.425798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.0.98.62:42067)
0927 02:44:27.425931 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.425936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.426893 (+ 957us) server_negotiation.cc:407] Connection header received
0927 02:44:27.428026 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.428032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.428086 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.428187 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.429048 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.429564 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.430738 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.431074 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.431150 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.433732 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.433736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.433739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.433766 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.435537 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.436249 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.436252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.436254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.436340 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.436753 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.436757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.436758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.437024 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.437096 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.437663 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.437782 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:44:27.438728 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.426268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:33646)
0927 02:44:27.426717 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.426783 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.427775 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.428315 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.428320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.428688 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.428914 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.428926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.429725 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.429729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.430627 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.430635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.431198 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.431202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.431365 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.432002 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.432016 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.433596 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.435659 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.435666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.435673 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.436079 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.436445 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.436450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.436455 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.436640 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.437137 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.437144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.437412 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.437580 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.437872 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"spinlock_wait_cycles":3968,"thread_start_us":130,"threads_started":1}
I20260927 02:44:27.438735 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.427074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.0.98.61:39225)
0927 02:44:27.427385 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.427395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.427434 (+ 39us) server_negotiation.cc:407] Connection header received
0927 02:44:27.427775 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.427795 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.428048 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.428215 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.429071 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.430117 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.430997 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.431439 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.431548 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.434451 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.434457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.434469 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.434496 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.436252 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.436758 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.436764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.436766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.436836 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.437196 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.437198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.437200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.437413 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.437574 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.437831 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.437967 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
I20260927 02:44:27.440219 7036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:44:27.447434 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.440609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.0.1:39092)
0927 02:44:27.440709 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.440724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.440787 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.441212 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.441214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.441328 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.441412 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.441416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.442069 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.442071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.442554 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.442560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.442999 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.443002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.443080 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.443457 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.443467 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.444650 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.446184 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.446187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.446188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.446378 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.446644 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.446646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.446648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.446736 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.447094 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.447096 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.447162 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.447244 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.447329 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 02:44:27.447527 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.440952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.0.98.61:39225)
0927 02:44:27.441050 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.441053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.441064 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:44:27.441089 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.441091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.441120 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.441182 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.441506 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.441979 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.442643 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.442877 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.442953 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.444764 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.444767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.444769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.444788 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.446084 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.446465 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.446467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.446468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.446539 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.446843 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.446847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.446848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.447003 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.447086 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.447319 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.447405 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:44:27.450439 7036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Bootstrap starting.
I20260927 02:44:27.452548 7036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:27.453212 7036 log.cc:824] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:27.454833 7036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: No bootstrap required, opened a new log
I20260927 02:44:27.457599 7036 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } }
I20260927 02:44:27.457799 7036 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:27.457875 7036 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b216448cab4b15b3e3f75ac5005c2e, State: Initialized, Role: FOLLOWER
I20260927 02:44:27.458271 7036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } }
I20260927 02:44:27.458707 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } } }
I20260927 02:44:27.458808 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:27.459303 7036 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:44:27.463446 7053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:44:27.463531 7053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:44:27.465360 6968 master_runner.cc:428] Detected that this master f4b216448cab4b15b3e3f75ac5005c2e is joining an existing cluster
I20260927 02:44:27.465408 6968 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:39225
I20260927 02:44:27.476076 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.467339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.0.98.61:39225)
0927 02:44:27.467468 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.467472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.467740 (+ 268us) server_negotiation.cc:407] Connection header received
0927 02:44:27.467847 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.467852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.467915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.467968 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.468685 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.469449 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.470381 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.470739 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.470858 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.472661 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.472665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.472668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.472688 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.474370 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.474892 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.474897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.474900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.474990 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.475274 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.475277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.475279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.475453 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.475518 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.475799 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.475936 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:27.476495 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.467422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.0.1:39096)
0927 02:44:27.467659 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.467670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.467751 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.468087 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.468090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.468356 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.468561 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.468570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.469576 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.469579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.470182 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.470191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.471102 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.471106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.471231 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.472064 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.472079 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.472537 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.474487 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.474492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.474498 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.474757 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.475087 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.475089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.475092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.475190 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.475574 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.475577 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.475659 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.476226 (+ 567us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.476335 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":57,"spinlock_wait_cycles":27776,"thread_start_us":83,"threads_started":1}
I20260927 02:44:27.476981 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.467262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.0.98.62:42067)
0927 02:44:27.467374 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.467379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.467647 (+ 268us) server_negotiation.cc:407] Connection header received
0927 02:44:27.467837 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.467841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.467913 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.467994 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.469592 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.470368 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.471375 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.471615 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.471736 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.473127 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.473130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.473132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.473154 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.474978 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.475429 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.475435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.475437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.475523 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.475861 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.475866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.475867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.476072 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.476201 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.476706 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.476855 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:44:27.477519 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.467186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:33656)
0927 02:44:27.467566 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.467581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.467679 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.468054 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.468057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.468358 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.468561 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.468570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.470477 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.470480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.471244 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.471252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.471742 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.471746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.471843 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.472298 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.472322 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.472691 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.475082 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.475088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.475092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.475326 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.475624 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.475629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.475631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.475721 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.476384 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.476386 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.476458 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.477161 (+ 703us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.477368 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":44,"spinlock_wait_cycles":896,"thread_start_us":75,"threads_started":1}
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:44:27.490727 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.481641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.0.98.61:39225)
0927 02:44:27.481746 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.481750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.482322 (+ 572us) server_negotiation.cc:407] Connection header received
0927 02:44:27.482481 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.482484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.482538 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.482617 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.483186 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.483915 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.484931 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.485320 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.485412 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.487645 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.487649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.487651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.487675 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.489067 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.489533 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.489538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.489539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.489614 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.489890 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.489893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.489895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.490046 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.490148 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.490470 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.490584 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:44:27.490743 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.481578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.0.1:39100)
0927 02:44:27.482215 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.482249 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.482368 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.482682 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.482687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.482989 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.483076 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.483082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.484007 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.484010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.484782 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.484806 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.485437 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.485441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.485548 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.486117 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.486136 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.487491 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.489202 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.489210 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.489216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.489434 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.489705 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.489710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.489712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.489794 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.490149 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.490156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.490268 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.490405 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.490589 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":89,"threads_started":1}
I20260927 02:44:27.492583 6504 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:39225: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 39225
}
attrs {
promote: true
}
}
dest_uuid: "ab504ffef5174fef83297fcd3dcf61db"
cas_config_opid_index: -1
I20260927 02:44:27.493274 6504 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } }
I20260927 02:44:27.495165 7064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f4b216448cab4b15b3e3f75ac5005c2e (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } }
I20260927 02:44:27.495810 7010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:27.497108 7064 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:39225
I20260927 02:44:27.497124 7010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Refusing update from remote peer ab504ffef5174fef83297fcd3dcf61db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 02:44:27.497499 6968 master.cc:537] Master@127.0.98.61:39225 shutting down...
I20260927 02:44:27.497814 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } }
I20260927 02:44:27.497946 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:27.498562 7064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:44:27.500808 6488 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
W20260927 02:44:27.501884 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db -> Peer f4b216448cab4b15b3e3f75ac5005c2e (127.0.98.61:39225): Couldn't send request to peer f4b216448cab4b15b3e3f75ac5005c2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:44:27.502892 6968 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:27.503149 6968 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:27.503253 6968 tablet_replica.cc:330] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 02:44:27.507819 6968 master.cc:552] Master@127.0.98.61:39225 shutdown complete.
I20260927 02:44:27.507910 6968 master_runner.cc:305] Clearing existing system tablet
I20260927 02:44:27.508127 6968 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:27.508194 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:27.509162 6968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:27.509922 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.510118 6968 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:27.510244 6968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:27.510285 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
uuid: "f4b216448cab4b15b3e3f75ac5005c2e"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "c36af7e7246a891b986406c122edd1ea"
server_key_iv: "9efcc5953a749affa3818df5f39b4993"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.511768 6968 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:27.513666 6968 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:44:27.513736 6968 log.cc:1196] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 02:44:27.514000 6968 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Deleting consensus metadata
I20260927 02:44:27.514160 6968 master_runner.cc:315] Copying system tablet from 127.0.98.62:42067
I20260927 02:44:27.515072 6968 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42067
I20260927 02:44:27.522897 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.515510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:33668)
0927 02:44:27.515730 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.515746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.515853 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.516170 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.516172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.516358 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.516444 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.516450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.517202 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.517208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.517806 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.517813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.518252 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.518256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.518347 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.519281 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.519293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.519596 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.521317 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.521321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.521324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.521524 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.521798 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.521802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.521804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.521894 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.522339 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.522341 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.522467 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.522571 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.522706 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260927 02:44:27.522886 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.515599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.0.98.62:42067)
0927 02:44:27.515729 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.515735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.515807 (+ 72us) server_negotiation.cc:407] Connection header received
0927 02:44:27.516018 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.516021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.516070 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.516154 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.516571 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.517095 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.517934 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.518145 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.518251 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.519734 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.519737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.519740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.519771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.521207 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.521622 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.521628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.521629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.521707 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.522008 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.522011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.522013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.522223 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.522282 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.522625 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.522730 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:44:27.523499 6524 tablet_copy_service.cc:140] P ab504ffef5174fef83297fcd3dcf61db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f4b216448cab4b15b3e3f75ac5005c2e ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33668)
I20260927 02:44:27.523659 6524 tablet_copy_service.cc:161] P ab504ffef5174fef83297fcd3dcf61db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f4b216448cab4b15b3e3f75ac5005c2e at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33668: session id = f4b216448cab4b15b3e3f75ac5005c2e-00000000000000000000000000000000
I20260927 02:44:27.525009 6524 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:44:27.526301 6968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:44:27.528697 6968 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: tablet copy: Starting download of 0 data blocks...
I20260927 02:44:27.528892 6968 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: tablet copy: Starting download of 1 WAL segments...
I20260927 02:44:27.530666 6968 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:44:27.532192 6524 tablet_copy_service.cc:342] P ab504ffef5174fef83297fcd3dcf61db: Request end of tablet copy session f4b216448cab4b15b3e3f75ac5005c2e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33668
I20260927 02:44:27.532300 6524 tablet_copy_service.cc:433] P ab504ffef5174fef83297fcd3dcf61db: ending tablet copy session f4b216448cab4b15b3e3f75ac5005c2e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f4b216448cab4b15b3e3f75ac5005c2e
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 02:44:27.535566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36705
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42067,127.0.98.61:39225,127.0.98.60:36705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:27.539942 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:27.541463 7079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.541491 7082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.541962 7080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:27.542176 6968 server_base.cc:1034] running on GCE node
I20260927 02:44:27.542347 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:27.542645 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:27.543790 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790477067543796 us; error 41 us; skew 500 ppm
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:27.545305 6968 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:44:27.545763 6968 webserver.cc:460] Webserver started at http://127.0.98.61:40187/ using document root <none> and password file <none>
I20260927 02:44:27.545959 6968 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:27.546017 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:27.547057 6968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:27.547959 7089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.548188 6968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:27.548281 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
uuid: "f4b216448cab4b15b3e3f75ac5005c2e"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "c36af7e7246a891b986406c122edd1ea"
server_key_iv: "9efcc5953a749affa3818df5f39b4993"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.548364 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:27.557602 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:27.559605 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:27.564213 6968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39225
I20260927 02:44:27.564673 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:44:27.565091 7141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39225 every 8 connection(s)
I20260927 02:44:27.566316 7142 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:44:27.567148 7142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Bootstrap starting.
I20260927 02:44:27.574981 7142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:27.575251 7142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: Bootstrap complete.
I20260927 02:44:27.575690 7142 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } }
I20260927 02:44:27.575819 7142 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f4b216448cab4b15b3e3f75ac5005c2e, State: Initialized, Role: LEARNER
I20260927 02:44:27.575943 7142 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } }
I20260927 02:44:27.576258 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } }
I20260927 02:44:27.576427 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: This master's current role is: LEARNER
I20260927 02:44:27.576463 7142 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:27.579909 7156 catalog_manager.cc:1259] Loaded cluster ID: e59f773e4b784f129d6a5bc439c8053a
I20260927 02:44:27.579986 7156 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: loading cluster ID for follower catalog manager: success
I20260927 02:44:27.581395 7156 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: acquiring CA information for follower catalog manager: success
I20260927 02:44:27.582386 7156 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 02:44:27.668128 7078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:27.668354 7078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:27.668392 7078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:27.672204 7078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:27.672246 7078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:27.672266 7078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:27.672282 7078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:27.672297 7078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:27.677680 7078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42067,127.0.98.61:39225,127.0.98.60:36705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:27.678792 7078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:27.679772 7078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:27.688086 7164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.688104 7161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.688050 7162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:27.688289 7078 server_base.cc:1034] running on GCE node
I20260927 02:44:27.688870 7078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:27.689898 7078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:27.691082 7078 hybrid_clock.cc:648] HybridClock initialized: now 1790477067691068 us; error 25 us; skew 500 ppm
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:27.693678 7078 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:44:27.694485 7078 webserver.cc:460] Webserver started at http://127.0.98.60:39041/ using document root <none> and password file <none>
I20260927 02:44:27.695060 7078 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:27.695145 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:27.695362 7078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:27.697331 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data/instance:
uuid: "a4d5f477b8834b3b91f2a5cdf923b67c"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "88f77f504384aee91df3c02a0472f290"
server_key_iv: "22796fb4b218236d4f86731b00150f18"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.697873 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal/instance:
uuid: "a4d5f477b8834b3b91f2a5cdf923b67c"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "88f77f504384aee91df3c02a0472f290"
server_key_iv: "22796fb4b218236d4f86731b00150f18"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.701324 7078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:44:27.703555 7171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.704766 7078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:27.704896 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
uuid: "a4d5f477b8834b3b91f2a5cdf923b67c"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "88f77f504384aee91df3c02a0472f290"
server_key_iv: "22796fb4b218236d4f86731b00150f18"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.705008 7078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:27.736099 7078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:27.738938 7078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:27.739138 7078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:27.746378 7078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36705
I20260927 02:44:27.746660 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36705 every 8 connection(s)
I20260927 02:44:27.747357 7078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:44:27.750324 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:27.752301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 7078
I20260927 02:44:27.754915 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:27.767278 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.757444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.0.98.62:42067)
0927 02:44:27.757562 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.757566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.757747 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:44:27.757920 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.757926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.757987 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.758058 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.759249 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.759993 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.760857 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.761261 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.761387 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.763839 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.763842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.763870 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.763929 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.765548 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.765981 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.765986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.765988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.766070 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.766347 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.766352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.766354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.766532 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.766636 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.766973 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.767088 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:44:27.768049 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.757224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:33670)
0927 02:44:27.757609 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.757655 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.757771 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.758124 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.758129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.758590 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.759108 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.759115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.760119 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.760121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.760739 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.760747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.761420 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.761422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.761499 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.762025 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.762044 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.763696 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.765645 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.765655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.765661 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.765871 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.766166 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.766169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.766171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.766257 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.766634 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.766640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.766780 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.766918 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.767224 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":97,"thread_start_us":144,"threads_started":1}
I20260927 02:44:27.768054 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.753882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.0.98.60:36705)
0927 02:44:27.754340 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.754356 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.754511 (+ 155us) server_negotiation.cc:407] Connection header received
0927 02:44:27.755371 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.755387 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.755721 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.756097 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.756886 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.758410 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.759302 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.759749 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.759924 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.763033 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.763039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.763052 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.763077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.764780 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.765189 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.765194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.765201 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.765332 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.765715 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.765720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.765722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.766295 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.766514 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.766794 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.767137 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":287,"spinlock_wait_cycles":896,"thread_start_us":115,"threads_started":1}
I20260927 02:44:27.769302 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:27.773505 6695 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:44:27.777357 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.769770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.0.1:39114)
0927 02:44:27.769913 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.769926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.770024 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.770371 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.770373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.770493 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.770576 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.770580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.771418 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.771421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.772054 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.772062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.772459 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.772461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.772530 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.772886 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.772896 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.774104 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.775814 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.775820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.775825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.776059 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.776332 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.776336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.776338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.776467 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.776855 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.776857 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.776955 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.777061 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.777182 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 02:44:27.777407 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.769895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.98.61:39225)
0927 02:44:27.770175 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.770185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.770197 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:44:27.770232 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.770233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.770270 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.770341 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.770674 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.771326 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.772151 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.772369 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.772460 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.774237 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.774240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.774243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.774271 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.775705 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.776158 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.776162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.776164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.776239 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.776586 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.776589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.776591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.776758 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.776869 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.777122 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.777256 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260927 02:44:27.778182 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:44:27.785880 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.778664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.0.1:45500)
0927 02:44:27.778770 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.778783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.778853 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.779066 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.779068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.779187 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.779250 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.779254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.780331 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.780333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.780825 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.780832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.781213 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.781215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.781271 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.781706 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.781719 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.782909 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.784437 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.784443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.784446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.784634 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.784895 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.784901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.784904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.784982 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.785327 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.785330 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.785408 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.785525 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.785654 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 02:44:27.785945 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.778744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.0.98.60:36705)
0927 02:44:27.778843 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.778847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.778860 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:27.778931 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.778934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.778970 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.779057 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.779338 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.780245 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.780918 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.781126 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.781198 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.783059 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.783063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.783065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.783091 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.784331 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.784727 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.784734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.784735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.784804 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.785068 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.785074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.785076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.785230 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.785327 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.785566 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.785709 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":40}
I20260927 02:44:27.789026 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Bootstrap starting.
I20260927 02:44:27.791157 7224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:27.791832 7224 log.cc:824] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:27.793802 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: No bootstrap required, opened a new log
I20260927 02:44:27.796868 7224 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } }
I20260927 02:44:27.797067 7224 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:27.797101 7224 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d5f477b8834b3b91f2a5cdf923b67c, State: Initialized, Role: FOLLOWER
I20260927 02:44:27.797454 7224 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } }
I20260927 02:44:27.797884 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } } }
I20260927 02:44:27.798048 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:27.798564 7224 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:44:27.803092 7241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:44:27.803172 7241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:44:27.806401 7078 master_runner.cc:428] Detected that this master a4d5f477b8834b3b91f2a5cdf923b67c is joining an existing cluster
I20260927 02:44:27.806452 7078 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36705
I20260927 02:44:27.817535 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.809088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:33678)
0927 02:44:27.809409 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.809432 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.809565 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.809939 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.809942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.810135 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.810242 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.810248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.811215 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.811218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.812328 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.812338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.813020 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.813023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.813452 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.813998 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.814009 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.814356 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.816221 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.816224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.816225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.816408 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.816686 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.816689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.816690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.816764 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.817146 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.817148 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.817219 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.817297 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.817391 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":95,"thread_start_us":82,"threads_started":1}
I20260927 02:44:27.817639 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.808870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.0.98.62:42067)
0927 02:44:27.808993 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.808999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.809502 (+ 503us) server_negotiation.cc:407] Connection header received
0927 02:44:27.809674 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.809677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.809732 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.809785 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.810370 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.811086 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.812466 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.812799 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.812873 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.814493 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.814499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.814503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.814531 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.816124 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.816508 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.816511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.816513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.816600 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.816849 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.816852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.816853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.817058 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.817154 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.817372 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.817488 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:44:27.821249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 6566
I20260927 02:44:27.828018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33107
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42067
--webserver_port=39669
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225,127.0.98.60:36705 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:44:27.836560 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.808523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.0.1:45504)
0927 02:44:27.809358 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.809371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.809484 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.809766 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.809769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.810010 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.810120 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.810128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.811100 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.811103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.812063 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.812073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.812732 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.812736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.812877 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.813425 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.813440 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.832840 (+ 19400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.835004 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.835007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.835009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.835272 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.835566 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.835569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.835571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.835638 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.836052 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.836055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.836128 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.836221 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.836366 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":46,"thread_start_us":104,"threads_started":1}
I20260927 02:44:27.837366 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.808551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.0.98.60:36705)
0927 02:44:27.808851 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.808856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.809434 (+ 578us) server_negotiation.cc:407] Connection header received
0927 02:44:27.809570 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.809573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.809606 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.809649 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.810238 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.810982 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.812181 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.812511 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.812589 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.833228 (+ 20639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.833232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.833235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.833262 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.834875 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.835410 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.835414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.835415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.835484 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.835715 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.835718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.835720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.835939 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.836022 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.837089 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.837196 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":234}
I20260927 02:44:27.838599 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.808479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.0.1:39124)
0927 02:44:27.808803 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.808820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.808920 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.809285 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.809289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.809572 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.809773 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.809782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.811510 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.811516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.812880 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.812888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.813588 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.813594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.813700 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.832416 (+ 18716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.832432 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.833977 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.836651 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.836657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.836662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.836924 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.837270 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.837275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.837278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.837377 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.837935 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.837940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.838055 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.838174 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.838336 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":44,"thread_start_us":83,"threads_started":1}
I20260927 02:44:27.839293 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.808937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.0.98.61:39225)
0927 02:44:27.809036 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.809040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.809053 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:27.809093 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.809096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.809147 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.809252 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.809898 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.810730 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.813011 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.813355 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.813433 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.834465 (+ 21032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.834471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.834477 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.834520 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.836508 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.837056 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.837060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.837065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.837156 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.837511 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.837516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.837521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.837787 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.837888 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.838925 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.839068 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:44:27.840910 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.821340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.0.1:45514)
0927 02:44:27.821521 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.821536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.821608 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.824880 (+ 3272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.824886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.825115 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.825250 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.825260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.826409 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.826415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.827408 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.827422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.828053 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.828058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.828181 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.828774 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.828790 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.830284 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.838985 (+ 8701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.838988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.838991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.839245 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.839765 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.839768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.839770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.839867 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.840332 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.840337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.840445 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.840536 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.840702 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 02:44:27.841514 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.821397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.0.98.60:36705)
0927 02:44:27.824594 (+ 3197us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.824602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.824616 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:27.824683 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.824686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.824738 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.824833 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.825376 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.826270 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.827529 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.827926 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.828029 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.830461 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.830464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.830467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.830490 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.838858 (+ 8368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.839561 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.839564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.839565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.839655 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.839996 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.839999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.840001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.840211 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.841168 (+ 957us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.841238 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.841361 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3130,"thread_start_us":87,"threads_started":1}
I20260927 02:44:27.842324 6504 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36705: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36705
}
attrs {
promote: true
}
}
dest_uuid: "ab504ffef5174fef83297fcd3dcf61db"
cas_config_opid_index: 5
I20260927 02:44:27.842950 6504 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } }
I20260927 02:44:27.844436 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.844687 7198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:27.845851 7066 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36705
I20260927 02:44:27.846052 7198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Refusing update from remote peer ab504ffef5174fef83297fcd3dcf61db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:44:27.846045 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.846132 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:27.847357 7078 master.cc:537] Master@127.0.98.60:36705 shutting down...
I20260927 02:44:27.848515 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:44:27.850397 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db -> Peer a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60:36705): Couldn't send request to peer a4d5f477b8834b3b91f2a5cdf923b67c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:44:27.852361 7078 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:27.852641 7078 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:27.852743 7078 tablet_replica.cc:330] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: stopping tablet replica
I20260927 02:44:27.856525 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.846912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.0.1:39138)
0927 02:44:27.847080 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.847095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.847173 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.851347 (+ 4174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.851350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.851368 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:27.851482 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.851488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.852774 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.852838 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.854206 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.854216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.856070 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.856073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.856169 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.856215 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.856291 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.856366 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 02:44:27.856973 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.847787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.0.98.61:39225)
0927 02:44:27.850951 (+ 3164us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.850955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.850969 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:27.851017 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.851020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.851076 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.851198 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:27.851609 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.852617 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.854346 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.855746 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.855849 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.856008 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.856752 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.856822 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3084}
I20260927 02:44:27.857764 7116 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 LEARNER]: Refusing update from remote peer ab504ffef5174fef83297fcd3dcf61db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:44:27.857863 7078 master.cc:552] Master@127.0.98.60:36705 shutdown complete.
I20260927 02:44:27.857901 7078 master_runner.cc:305] Clearing existing system tablet
I20260927 02:44:27.858126 7078 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:27.858177 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:27.858222 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 02:44:27.860021 7116 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.862185 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab504ffef5174fef83297fcd3dcf61db. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.862288 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: This master's current role is: LEARNER
I20260927 02:44:27.862370 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: true } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.862444 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: This master's current role is: LEARNER
I20260927 02:44:27.863229 7066 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: attempting to promote NON_VOTER f4b216448cab4b15b3e3f75ac5005c2e to VOTER
I20260927 02:44:27.863678 7066 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } }
I20260927 02:44:27.864254 7078 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.000s
I20260927 02:44:27.865130 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.864829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.0.1:45518)
0927 02:44:27.864990 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.865060 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 02:44:27.865120 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.865398 7078 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:27.865505 7078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:27.865536 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
uuid: "a4d5f477b8834b3b91f2a5cdf923b67c"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "88f77f504384aee91df3c02a0472f290"
server_key_iv: "22796fb4b218236d4f86731b00150f18"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.865819 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.865603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.0.1:45520)
0927 02:44:27.865719 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.865761 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 02:44:27.865921 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db -> Peer a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60:36705): Couldn't send request to peer a4d5f477b8834b3b91f2a5cdf923b67c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:44:27.866520 7116 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 LEARNER]: Refusing update from remote peer ab504ffef5174fef83297fcd3dcf61db: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 02:44:27.866854 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 02:44:27.867501 7078 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:27.868381 7063 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f4b216448cab4b15b3e3f75ac5005c2e (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.869765 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.869858 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:27.869848 7078 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:44:27.869920 7078 log.cc:1196] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 02:44:27.870144 7078 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Deleting consensus metadata
I20260927 02:44:27.870283 7078 master_runner.cc:315] Copying system tablet from 127.0.98.62:42067
I20260927 02:44:27.870409 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.870515 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:27.871105 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f4b216448cab4b15b3e3f75ac5005c2e (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.871405 7078 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42067
I20260927 02:44:27.872705 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.872835 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:27.881378 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.872164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:33690)
0927 02:44:27.873198 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:27.873212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:27.873318 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:27.873589 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:27.873592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:27.873802 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:27.873913 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.873920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.874980 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.874984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.875967 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:27.875978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.876535 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.876539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:27.876627 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.877254 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:27.877268 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:27.877690 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:27.879773 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.879776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.879778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.880054 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.880352 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:27.880355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:27.880357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:27.880446 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:27.880866 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:27.880869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:27.880963 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 02:44:27.881056 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:44:27.881184 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"thread_start_us":73,"threads_started":1}
I20260927 02:44:27.881390 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:27.871955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.98.62:42067)
0927 02:44:27.872062 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:27.872067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:27.873289 (+ 1222us) server_negotiation.cc:407] Connection header received
0927 02:44:27.873408 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:27.873411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:27.873467 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:27.873551 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:27.874029 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.874838 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.876099 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:27.876412 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:27.876522 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:27.877821 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:27.877826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:27.877830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:27.877855 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:27.879661 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.880164 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.880167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.880169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.880256 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:27.880548 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:27.880551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:27.880553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:27.880770 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:27.880866 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:27.881146 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:44:27.881256 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:44:27.882081 6524 tablet_copy_service.cc:140] P ab504ffef5174fef83297fcd3dcf61db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a4d5f477b8834b3b91f2a5cdf923b67c ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33690)
I20260927 02:44:27.882177 6524 tablet_copy_service.cc:161] P ab504ffef5174fef83297fcd3dcf61db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a4d5f477b8834b3b91f2a5cdf923b67c at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33690: session id = a4d5f477b8834b3b91f2a5cdf923b67c-00000000000000000000000000000000
I20260927 02:44:27.883253 6524 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:44:27.884500 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:44:27.886730 7078 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: tablet copy: Starting download of 0 data blocks...
I20260927 02:44:27.886916 7078 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: tablet copy: Starting download of 1 WAL segments...
I20260927 02:44:27.888490 7078 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:44:27.890146 6524 tablet_copy_service.cc:342] P ab504ffef5174fef83297fcd3dcf61db: Request end of tablet copy session a4d5f477b8834b3b91f2a5cdf923b67c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33690
I20260927 02:44:27.890226 6524 tablet_copy_service.cc:433] P ab504ffef5174fef83297fcd3dcf61db: ending tablet copy session a4d5f477b8834b3b91f2a5cdf923b67c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a4d5f477b8834b3b91f2a5cdf923b67c
I20260927 02:44:27.898028 7078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:27.899662 7262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.899724 7263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.899751 7265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:27.900193 7078 server_base.cc:1034] running on GCE node
I20260927 02:44:27.900396 7078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:27.900669 7078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:27.901815 7078 hybrid_clock.cc:648] HybridClock initialized: now 1790477067901791 us; error 40 us; skew 500 ppm
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:27.903404 7078 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:44:27.903941 7078 webserver.cc:460] Webserver started at http://127.0.98.60:38399/ using document root <none> and password file <none>
I20260927 02:44:27.904151 7078 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:27.904191 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:27.905473 7078 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:27.906441 7272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:27.906728 7078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:27.906805 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
uuid: "a4d5f477b8834b3b91f2a5cdf923b67c"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "88f77f504384aee91df3c02a0472f290"
server_key_iv: "22796fb4b218236d4f86731b00150f18"
server_key_version: "encryptionkey@0"
I20260927 02:44:27.906884 7078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:27.919649 7078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:27.921841 7078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:27.926664 7078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36705
I20260927 02:44:27.927003 7324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36705 every 8 connection(s)
I20260927 02:44:27.927260 7078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:44:27.929049 7325 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:44:27.930109 7325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Bootstrap starting.
I20260927 02:44:27.941294 7325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:27.941614 7325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Bootstrap complete.
I20260927 02:44:27.942194 7325 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } }
I20260927 02:44:27.942344 7325 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4d5f477b8834b3b91f2a5cdf923b67c, State: Initialized, Role: LEARNER
I20260927 02:44:27.942466 7325 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } }
I20260927 02:44:27.942798 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:27.942929 7325 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:27.943004 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: This master's current role is: LEARNER
I20260927 02:44:27.947146 7339 catalog_manager.cc:1259] Loaded cluster ID: e59f773e4b784f129d6a5bc439c8053a
I20260927 02:44:27.947211 7339 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: loading cluster ID for follower catalog manager: success
I20260927 02:44:27.949378 7339 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: acquiring CA information for follower catalog manager: success
I20260927 02:44:27.950794 7339 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 02:44:27.969527 7251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:27.969751 7251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:27.969791 7251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:27.973815 7251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:27.973872 7251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:27.973965 7251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:27.979609 7251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39669
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225,127.0.98.60:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:27.980639 7251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:27.981593 7251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:27.991199 7344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.991225 7345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:27.991216 7347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:27.991528 7251 server_base.cc:1034] running on GCE node
I20260927 02:44:27.992174 7251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:27.992784 7251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:27.993978 7251 hybrid_clock.cc:648] HybridClock initialized: now 1790477067993955 us; error 31 us; skew 500 ppm
Sep 27 02:44:27 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:27.996622 7251 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:44:27.997489 7251 webserver.cc:460] Webserver started at http://127.0.98.1:39669/ using document root <none> and password file <none>
I20260927 02:44:27.997993 7251 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:27.998061 7251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:27.999847 6829 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
I20260927 02:44:28.002138 7251 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 02:44:28.004997 7354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.006656 7251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:44:28.006820 7251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "9d39bf310b3a486bae2e100f6115f8f9"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "4f8e4158c932370c546793adb27f5b5c"
server_key_iv: "15605ddab1d6ceb528e0c52f68bd670d"
server_key_version: "encryptionkey@0"
I20260927 02:44:28.007509 7251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:28.022535 7251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:28.025555 7251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:28.025744 7251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:28.026305 7251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:28.027156 7251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:28.027227 7251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.027295 7251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:28.027347 7251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.037317 7251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33107
I20260927 02:44:28.037336 7467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33107 every 8 connection(s)
I20260927 02:44:28.038270 7251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:28.047089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 7251
I20260927 02:44:28.047302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 6700
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:44:28.054062 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.040029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53223 (local address 127.0.98.61:39225)
0927 02:44:28.040156 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.040161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.041608 (+ 1447us) server_negotiation.cc:407] Connection header received
0927 02:44:28.042347 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.042351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.042436 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.042522 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.044193 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.044769 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.045923 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.046139 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.046255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.049416 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.049421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.049424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.049455 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.051274 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.051938 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.051943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.051945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.052032 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.052444 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.052450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.052455 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.052692 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.052804 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.053752 (+ 948us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.053878 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:44:28.054207 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.040796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35669 (local address 127.0.98.62:42067)
0927 02:44:28.040939 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.040945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.042623 (+ 1678us) server_negotiation.cc:407] Connection header received
0927 02:44:28.042785 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.042787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.042881 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.042977 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.044208 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.045020 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.046206 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.046516 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.046610 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.049999 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.050005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.050011 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.050044 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.052283 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.052977 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.052981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.052982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.053061 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.053351 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.053354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.053356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.053558 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.053627 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.053964 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.054069 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:28.055016 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.040455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.98.1:53223)
0927 02:44:28.041510 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.041523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.042111 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.042716 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.042727 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.043272 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.044044 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.044063 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.044917 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.044922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.045788 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.045797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.046398 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.046402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.046594 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.047236 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.047257 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.049226 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.051412 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.051419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.051434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.051769 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.052144 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.052147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.052152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.052339 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.052858 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.052865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.053176 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.053355 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.053958 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":175,"spinlock_wait_cycles":18432,"thread_start_us":149,"threads_started":1}
I20260927 02:44:28.055006 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.041132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:35669)
0927 02:44:28.042540 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.042553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.042684 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.043013 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.043015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.043272 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.044052 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.044065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.045155 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.045157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.046090 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.046099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.046742 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.046746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.046867 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.047331 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.048212 (+ 881us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.049830 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.052392 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.052395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.052397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.052873 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.053168 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.053170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.053172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.053246 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.053705 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.053708 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.053785 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.054608 (+ 823us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.054723 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"mutex_wait_us":169,"spinlock_wait_cycles":2304,"thread_start_us":96,"threads_started":1}
I20260927 02:44:28.056305 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39225
I20260927 02:44:28.056533 7470 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.056703 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.040455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.98.1:33421)
0927 02:44:28.041203 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.041267 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.042113 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.043402 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.043404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.043556 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.044056 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.044065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.045375 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.045378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.046194 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.046203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.046625 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.046629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.046725 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.048264 (+ 1539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.048349 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.051296 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.053577 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.053583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.053588 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.053857 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.055654 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.055657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.055659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.055740 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.056181 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.056184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.056270 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.056374 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.056507 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":47,"thread_start_us":122,"threads_started":1}
I20260927 02:44:28.056726 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.040029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33421 (local address 127.0.98.60:36705)
0927 02:44:28.040269 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.040274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.041368 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 02:44:28.042361 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.042365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.042420 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.042522 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.044208 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.045240 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.046278 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.046516 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.046629 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.051473 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.051479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.051486 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.051520 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.053425 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.055435 (+ 2010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.055441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.055444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.055540 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.055838 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.055841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.055843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.056079 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.056190 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.056439 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.056567 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260927 02:44:28.057145 7470 heartbeater.cc:511] Master 127.0.98.61:39225 requested a full tablet report, sending...
I20260927 02:44:28.057451 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36705
I20260927 02:44:28.057530 7469 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.057691 7469 heartbeater.cc:511] Master 127.0.98.60:36705 requested a full tablet report, sending...
I20260927 02:44:28.058316 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260927 02:44:28.058411 7468 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.058588 7468 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
I20260927 02:44:28.058662 7106 ts_manager.cc:194] Registered new tserver with Master: 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107)
I20260927 02:44:28.058737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45807
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42067
--webserver_port=40835
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225,127.0.98.60:36705 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:28.059378 7289 ts_manager.cc:194] Registered new tserver with Master: 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107)
I20260927 02:44:28.060364 6504 ts_manager.cc:194] Re-registered known tserver with Master: 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107)
I20260927 02:44:28.060956 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35669
W20260927 02:44:28.170784 7476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:28.171021 7476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:28.171087 7476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:28.175343 7476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:28.175398 7476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:28.175483 7476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:28.180847 7476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40835
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225,127.0.98.60:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:28.181787 7476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:28.182837 7476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:28.192265 7481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:28.192256 7476 server_base.cc:1034] running on GCE node
W20260927 02:44:28.192219 7484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:28.192221 7482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:28.192835 7476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:28.193442 7476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:28.194604 7476 hybrid_clock.cc:648] HybridClock initialized: now 1790477068194592 us; error 23 us; skew 500 ppm
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477068, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:28.197307 7476 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:44:28.198110 7476 webserver.cc:460] Webserver started at http://127.0.98.2:40835/ using document root <none> and password file <none>
I20260927 02:44:28.198576 7476 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:28.198643 7476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:28.202417 7476 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:44:28.204474 7491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.205710 7476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:28.205834 7476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "26e71a93b4414e3bb237b4bbf92be938"
format_stamp: "Formatted at 2026-09-27 02:44:26 on dist-test-slave-gkrl"
server_key: "e6e72ce2abe6996b6f4451c29c3c2760"
server_key_iv: "ea78790e161a78d17eff0f56045cc1a5"
server_key_version: "encryptionkey@0"
I20260927 02:44:28.206382 7476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:28.222694 6963 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
I20260927 02:44:28.225229 7476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:28.228263 7476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:28.228446 7476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:28.229007 7476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:28.229825 7476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:28.229869 7476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.229931 7476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:28.229974 7476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.238741 7476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45807
I20260927 02:44:28.238812 7604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45807 every 8 connection(s)
I20260927 02:44:28.239753 7476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:28.245244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 7476
I20260927 02:44:28.245451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 6834
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477068, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477068, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:44:28.253211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35271
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42067
--webserver_port=45939
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225,127.0.98.60:36705 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477068, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:44:28.258248 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.243773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51043 (local address 127.0.98.60:36705)
0927 02:44:28.243912 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.243917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.244178 (+ 261us) server_negotiation.cc:407] Connection header received
0927 02:44:28.245052 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.245058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.245114 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.245190 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.246686 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.247422 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.249105 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.249481 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.249562 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.253529 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.253537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.253540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.253564 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.255715 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.256703 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.256707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.256709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.256793 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.257170 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.257173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.257175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.257387 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.257466 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.257934 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.258068 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:44:28.258734 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.243422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34551 (local address 127.0.98.61:39225)
0927 02:44:28.243567 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.243574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.243740 (+ 166us) server_negotiation.cc:407] Connection header received
0927 02:44:28.244786 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.244789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.244835 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.244919 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.246577 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.247446 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.250319 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.250765 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.250888 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.255014 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.255020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.255024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.255055 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.256816 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.257275 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.257278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.257280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.257351 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.257651 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.257654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.257656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.257883 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.257965 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.258492 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.258591 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 02:44:28.259277 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.243682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.98.2:51043)
0927 02:44:28.244067 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.244082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.244220 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.245304 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.245307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.246444 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.246556 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.246562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.247570 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.247573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.248972 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.248982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.249748 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.249751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.250008 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.250877 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.250895 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.253309 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.256346 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.256352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.256355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.256584 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.256916 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.256918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.256920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.257058 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.257518 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.257520 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.257646 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.258391 (+ 745us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.258517 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":293,"spinlock_wait_cycles":7680,"thread_start_us":101,"threads_started":1}
I20260927 02:44:28.259289 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.243266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.98.2:34551)
0927 02:44:28.243616 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.243631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.244243 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.244998 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.245002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.246267 (+ 1265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.246431 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.246442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.247596 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.247602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.250187 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.250200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.251025 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.251031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.251172 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.251730 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.251748 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.254859 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.256916 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.256918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.256920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.257174 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.257467 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.257471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.257474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.257552 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.258006 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.258008 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.258105 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.258188 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.258309 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":1285,"spinlock_wait_cycles":896,"thread_start_us":118,"threads_started":1}
I20260927 02:44:28.259585 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.241935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:33629)
0927 02:44:28.242870 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.242913 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.244220 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.244844 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.244853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.245589 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.246207 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.246228 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.247353 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.247356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.248972 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.248982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.249720 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.249725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.250010 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.250782 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.250806 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.253839 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.255833 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.255841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.255865 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.256235 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.256581 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.256587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.256593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.256779 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.257312 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.257320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.257656 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.257854 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.258213 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":172,"spinlock_wait_cycles":8576,"thread_start_us":594,"threads_started":1}
I20260927 02:44:28.260236 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.241545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33629 (local address 127.0.98.62:42067)
0927 02:44:28.241698 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.241702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.243032 (+ 1330us) server_negotiation.cc:407] Connection header received
0927 02:44:28.244499 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.244506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.244567 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.244662 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.246374 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.247197 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.249105 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.249443 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.249525 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.253989 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.253996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.254002 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.254039 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.255715 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.256371 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.256377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.256382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.256471 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.256892 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.256898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.256903 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.257150 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.257252 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.259867 (+ 2615us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.260012 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 02:44:28.260466 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36705
I20260927 02:44:28.260699 7606 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.260799 7605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260927 02:44:28.260886 7605 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.261241 7606 heartbeater.cc:511] Master 127.0.98.60:36705 requested a full tablet report, sending...
I20260927 02:44:28.261267 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39225
I20260927 02:44:28.261234 7605 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
I20260927 02:44:28.261412 7607 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.261595 7607 heartbeater.cc:511] Master 127.0.98.61:39225 requested a full tablet report, sending...
I20260927 02:44:28.262349 6504 ts_manager.cc:194] Re-registered known tserver with Master: 26e71a93b4414e3bb237b4bbf92be938 (127.0.98.2:45807)
I20260927 02:44:28.262625 7106 ts_manager.cc:194] Registered new tserver with Master: 26e71a93b4414e3bb237b4bbf92be938 (127.0.98.2:45807)
I20260927 02:44:28.263062 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33629
I20260927 02:44:28.263585 7289 ts_manager.cc:194] Registered new tserver with Master: 26e71a93b4414e3bb237b4bbf92be938 (127.0.98.2:45807)
I20260927 02:44:28.297873 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.293950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.0.98.60:36705)
0927 02:44:28.294169 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.294173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.294299 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:44:28.294441 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.294443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.294494 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.294578 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:28.294864 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.295447 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.296867 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.297301 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.297390 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.297500 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.297646 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.297701 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 02:44:28.297897 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.293975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.0.1:45536)
0927 02:44:28.294209 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.294226 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.294342 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.294608 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.294611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.294629 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:28.294728 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.294734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.295562 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.295565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.296739 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.296751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.297430 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.297434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.297521 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.297545 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.297622 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.297724 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 02:44:28.298671 7299 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 02:44:28.299028 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab504ffef5174fef83297fcd3dcf61db. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:28.299172 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: This master's current role is: LEARNER
W20260927 02:44:28.366943 7612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:28.367183 7612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:28.367223 7612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:28.371331 7612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:28.371390 7612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:28.371447 7612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:28.376674 7612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45939
--tserver_master_addrs=127.0.98.62:42067,127.0.98.61:39225,127.0.98.60:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:28.377620 7612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:28.378693 7612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:28.387874 7620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:28.387957 7618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:28.388000 7617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:28.388212 7612 server_base.cc:1034] running on GCE node
I20260927 02:44:28.388639 7612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:28.389292 7612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:28.390467 7612 hybrid_clock.cc:648] HybridClock initialized: now 1790477068390453 us; error 27 us; skew 500 ppm
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477068, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:28.393133 7612 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:44:28.393889 7612 webserver.cc:460] Webserver started at http://127.0.98.3:45939/ using document root <none> and password file <none>
I20260927 02:44:28.394390 7612 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:28.394454 7612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:28.398200 7612 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:28.400303 7627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.401559 7612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:28.401649 7612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "5162b030e0504b25a9388e542c12c2a8"
format_stamp: "Formatted at 2026-09-27 02:44:27 on dist-test-slave-gkrl"
server_key: "6d3ced3c0a1135d0f723645dbb9d0d0b"
server_key_iv: "c4125df49c8a6397a0c10c9703d28b1d"
server_key_version: "encryptionkey@0"
I20260927 02:44:28.402148 7612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:28.414040 7612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:28.416873 7612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:28.417104 7612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:28.417650 7612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:28.418426 7612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:28.418471 7612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.418536 7612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:28.418566 7612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:28.427655 7612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35271
I20260927 02:44:28.427695 7740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35271 every 8 connection(s)
I20260927 02:44:28.428766 7612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:28.429983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 7612
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477068, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477068, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:44:28 dist-test-slave-gkrl krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477068, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:44:28.447377 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.432052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54399 (local address 127.0.98.62:42067)
0927 02:44:28.432182 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.432187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.432275 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:44:28.433601 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.433607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.433658 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.433742 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.435725 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.436553 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.438373 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.438693 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.438764 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.441959 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.441962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.441965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.441989 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.443980 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.445488 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.445492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.445494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.445578 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.446017 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.446023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.446027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.446256 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.446336 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.447066 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.447182 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:44:28.447760 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.433934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.0.98.62:42067)
0927 02:44:28.434648 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.434654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.435591 (+ 937us) server_negotiation.cc:407] Connection header received
0927 02:44:28.435644 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.435647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.435701 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.435795 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.436755 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.437584 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.438757 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.439135 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.439244 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.441400 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.441406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.441410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.441440 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.444509 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.445796 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.445802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.445804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.445899 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.446245 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.446248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.446250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.446509 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.446649 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.447408 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.447558 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":82,"threads_started":1}
I20260927 02:44:28.449177 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.431389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38515 (local address 127.0.98.61:39225)
0927 02:44:28.431514 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.431519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.433941 (+ 2422us) server_negotiation.cc:407] Connection header received
0927 02:44:28.434156 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.434160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.434218 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.434285 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.437523 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.438248 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.440503 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.440842 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.440921 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.445476 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.445479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.445482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.445507 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.447376 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.447909 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.447925 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.447930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.448009 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.448304 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.448307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.448308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.448506 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.448625 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.448942 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.449048 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:44:28.449803 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.431082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:54399)
0927 02:44:28.432110 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.432174 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.433325 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.434023 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.434032 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.434701 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.435559 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.435579 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.436732 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.436736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.438232 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.438243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.438948 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.438952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.439147 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.439850 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.439895 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.441744 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.444648 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.444654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.444671 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.445013 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.445701 (+ 688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.445707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.445712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.445896 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.446432 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.446440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.446769 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.448039 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.448450 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":214,"spinlock_wait_cycles":44672,"thread_start_us":120,"threads_started":1}
I20260927 02:44:28.450095 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.431191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.98.3:38515)
0927 02:44:28.433853 (+ 2662us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.433870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.434025 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.434466 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.434470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.434751 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.437017 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.437025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.438375 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.438378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.439155 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.439162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.441439 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.441442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.441526 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.442342 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.442363 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.445336 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.447497 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.447503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.447508 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.447767 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.448116 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.448118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.448120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.448206 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.448674 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.448677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.448759 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.449309 (+ 550us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.449737 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2589,"mutex_wait_us":2221,"thread_start_us":108,"threads_started":1}
I20260927 02:44:28.451040 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.432948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.0.98.60:36705)
0927 02:44:28.433150 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.433160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.435046 (+ 1886us) server_negotiation.cc:407] Connection header received
0927 02:44:28.435098 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.435102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.435151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.435238 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.436380 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.437294 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.438601 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.438977 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.439073 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.442092 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.442103 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.442109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.442142 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.447617 (+ 5475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.448177 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.448181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.448183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.448268 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.449028 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.449031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.449033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.449272 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.449411 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.450691 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.450843 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260927 02:44:28.451702 7743 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39225
I20260927 02:44:28.451964 7743 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.452414 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.431082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.98.3:40489)
0927 02:44:28.432110 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.432174 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.433503 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.434023 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.434032 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.434701 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:28.435559 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.435579 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.436732 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.436736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.439782 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.439795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.441077 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.441081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.441186 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.441767 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:28.441781 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:28.443950 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:28.446477 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.446480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.446483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.446741 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.448600 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:28.448607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:28.448610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:28.448710 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:28.449866 (+ 1156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:28.449870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:28.449975 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.452003 (+ 2028us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.452157 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":2234,"spinlock_wait_cycles":8704,"thread_start_us":132,"threads_started":1}
I20260927 02:44:28.452477 7743 heartbeater.cc:511] Master 127.0.98.61:39225 requested a full tablet report, sending...
I20260927 02:44:28.453204 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.430749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40489 (local address 127.0.98.60:36705)
0927 02:44:28.430904 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.430911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.432283 (+ 1372us) server_negotiation.cc:407] Connection header received
0927 02:44:28.433641 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.433645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.433694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.433747 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.435725 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.436558 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.439947 (+ 3389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.440279 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.440358 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.444133 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.444137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.444140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.444166 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.446174 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.448399 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.448402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.448404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.448474 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.449489 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.449493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.449494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.449706 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.449795 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.452763 (+ 2968us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.453029 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:44:28.453776 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36705
I20260927 02:44:28.453883 7742 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.454084 7742 heartbeater.cc:511] Master 127.0.98.60:36705 requested a full tablet report, sending...
I20260927 02:44:28.454166 7106 ts_manager.cc:194] Registered new tserver with Master: 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271)
I20260927 02:44:28.451366 6504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:44:28.454800 7289 ts_manager.cc:194] Registered new tserver with Master: 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271)
I20260927 02:44:28.455053 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260927 02:44:28.455165 7741 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:28.455338 7741 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
W20260927 02:44:28.455878 6504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:28.456054 6503 ts_manager.cc:194] Re-registered known tserver with Master: 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271)
I20260927 02:44:28.456815 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54399
I20260927 02:44:28.457126 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.433409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.0.98.61:39225)
0927 02:44:28.439792 (+ 6383us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.439798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.439814 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:28.439852 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.439858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.439970 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.443163 (+ 3193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.443281 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.444216 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.449771 (+ 5555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.451323 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.451776 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.453157 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.453161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.453164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.453189 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.455118 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.455670 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.455676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.455684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.455782 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.456148 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.456154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.456158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.456382 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.456507 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.456765 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.456884 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6295,"thread_start_us":76,"threads_started":1}
I20260927 02:44:28.458650 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: true } } }
I20260927 02:44:28.458809 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:28.461603 7760 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: attempting to promote NON_VOTER a4d5f477b8834b3b91f2a5cdf923b67c to VOTER
I20260927 02:44:28.461925 7760 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } }
I20260927 02:44:28.463165 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Refusing update from remote peer ab504ffef5174fef83297fcd3dcf61db: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 02:44:28.463554 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:44:28.464562 7299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 LEARNER]: Refusing update from remote peer ab504ffef5174fef83297fcd3dcf61db: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 02:44:28.465088 7759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:44:28.465591 7760 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:28.466138 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:28.467252 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:28.467221 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:28.467329 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:28.467568 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:28.467701 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:28.467581 7299 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:28.467337 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:28.469928 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:28.470038 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:28.471984 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab504ffef5174fef83297fcd3dcf61db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:28.472108 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:28.476027 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.469862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45807 (local address 127.0.0.1:39704)
0927 02:44:28.470535 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.470549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.470649 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.471378 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.471382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.471400 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:28.471505 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.471512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.472982 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.472988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.474789 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.474800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.475588 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.475591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.475672 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.475694 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.475756 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.475823 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":57,"thread_start_us":103,"threads_started":1}
I20260927 02:44:28.476049 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.470330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.0.98.2:45807)
0927 02:44:28.470905 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.470915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.470934 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:44:28.471003 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.471008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.471235 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.471416 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:28.471648 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.472842 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.474937 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.475465 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.475589 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.475743 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.475792 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.475877 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":108,"threads_started":1}
I20260927 02:44:28.476383 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.469257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33107 (local address 127.0.0.1:40896)
0927 02:44:28.469582 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.469599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.469700 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.471748 (+ 2048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.471753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.471774 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:28.471859 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.471864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.473353 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.473357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.474789 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.474800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.476018 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.476022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.476118 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.476141 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.476194 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.476258 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":127}
I20260927 02:44:28.477545 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.469556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.0.98.1:33107)
0927 02:44:28.471339 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.471347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.471366 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:44:28.471433 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.471437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.471607 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.471762 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:28.472012 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.473205 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.474919 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.475789 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.477078 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.477213 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.477267 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.477366 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1701,"thread_start_us":91,"threads_started":1}
I20260927 02:44:28.477700 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.469950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35271 (local address 127.0.0.1:35180)
0927 02:44:28.472308 (+ 2358us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.472321 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.472424 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.473021 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.473024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.473042 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:28.473124 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.473129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.474595 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.474601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.475802 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.475812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.477266 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.477270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.477374 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.477398 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.477476 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.477542 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2299,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260927 02:44:28.478091 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.470383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.98.3:35271)
0927 02:44:28.471729 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.471741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.472420 (+ 679us) server_negotiation.cc:407] Connection header received
0927 02:44:28.472543 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.472548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.472737 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.472892 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:28.473252 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.474475 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.476009 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.476884 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.477026 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.477199 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.477852 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.477948 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":102,"threads_started":1}
I20260927 02:44:28.479372 7539 tablet_service.cc:1467] Processing CreateTablet for tablet bdcf99af1b424dc89e961f6aec7aebaf (DEFAULT_TABLE table=test-table [id=862fe28913a44ea887d09deeefd479a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:28.480297 7539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcf99af1b424dc89e961f6aec7aebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:28.480387 7402 tablet_service.cc:1467] Processing CreateTablet for tablet bdcf99af1b424dc89e961f6aec7aebaf (DEFAULT_TABLE table=test-table [id=862fe28913a44ea887d09deeefd479a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:28.480973 7675 tablet_service.cc:1467] Processing CreateTablet for tablet bdcf99af1b424dc89e961f6aec7aebaf (DEFAULT_TABLE table=test-table [id=862fe28913a44ea887d09deeefd479a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:28.481443 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcf99af1b424dc89e961f6aec7aebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:28.481950 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcf99af1b424dc89e961f6aec7aebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:28.487321 7774 tablet_bootstrap.cc:492] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: Bootstrap starting.
I20260927 02:44:28.489351 7774 tablet_bootstrap.cc:654] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:28.489574 7775 tablet_bootstrap.cc:492] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Bootstrap starting.
I20260927 02:44:28.490334 7774 log.cc:824] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:28.490507 7776 tablet_bootstrap.cc:492] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: Bootstrap starting.
I20260927 02:44:28.491636 7775 tablet_bootstrap.cc:654] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:28.492352 7775 log.cc:824] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:28.492687 7776 tablet_bootstrap.cc:654] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:28.492759 7774 tablet_bootstrap.cc:492] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: No bootstrap required, opened a new log
I20260927 02:44:28.492977 7774 ts_tablet_manager.cc:1397] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:44:28.493502 7776 log.cc:824] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:28.494104 7775 tablet_bootstrap.cc:492] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: No bootstrap required, opened a new log
I20260927 02:44:28.494289 7775 ts_tablet_manager.cc:1397] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:44:28.495496 7776 tablet_bootstrap.cc:492] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: No bootstrap required, opened a new log
I20260927 02:44:28.495688 7776 ts_tablet_manager.cc:1397] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:44:28.497650 7775 raft_consensus.cc:348] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.497879 7775 raft_consensus.cc:374] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:28.497942 7775 raft_consensus.cc:729] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d39bf310b3a486bae2e100f6115f8f9, State: Initialized, Role: FOLLOWER
I20260927 02:44:28.498359 7775 consensus_queue.cc:260] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.498255 7774 raft_consensus.cc:348] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.498498 7774 raft_consensus.cc:374] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:28.498548 7774 raft_consensus.cc:729] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e71a93b4414e3bb237b4bbf92be938, State: Initialized, Role: FOLLOWER
I20260927 02:44:28.498999 7774 consensus_queue.cc:260] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.499312 7468 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
I20260927 02:44:28.499370 7775 ts_tablet_manager.cc:1428] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:44:28.500054 7774 ts_tablet_manager.cc:1428] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 02:44:28.500541 7605 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
I20260927 02:44:28.503224 7776 raft_consensus.cc:348] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.503531 7776 raft_consensus.cc:374] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:28.503597 7776 raft_consensus.cc:729] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5162b030e0504b25a9388e542c12c2a8, State: Initialized, Role: FOLLOWER
I20260927 02:44:28.504087 7776 consensus_queue.cc:260] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.505033 7776 ts_tablet_manager.cc:1428] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260927 02:44:28.506258 7741 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
W20260927 02:44:28.539820 7471 tablet.cc:2369] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:28.570731 7781 raft_consensus.cc:482] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:28.570937 7781 raft_consensus.cc:504] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.572160 7781 leader_election.cc:288] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107), 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271)
I20260927 02:44:28.576499 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.572647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35271 (local address 127.0.98.2:38895)
0927 02:44:28.572801 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.572813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.572890 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.573200 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.573203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.573218 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:28.573334 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.573339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.574268 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.574271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.575108 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.575117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.576009 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.576012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.576173 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.576203 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.576283 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.576353 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":3}
I20260927 02:44:28.576481 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.572855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38895 (local address 127.0.98.3:35271)
0927 02:44:28.572978 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.572996 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.573008 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:44:28.573052 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.573054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.573098 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.573175 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:28.573440 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.574155 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.575218 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.575839 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.576019 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.576091 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.576300 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.576363 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:44:28.576953 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.572383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33107 (local address 127.0.98.2:43767)
0927 02:44:28.572517 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:28.572534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:28.572621 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:28.573009 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:28.573012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:28.573032 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:28.573152 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.573162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.574183 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.574186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.575422 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:28.575431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.576503 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.576506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:28.576646 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.576681 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:44:28.576760 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:44:28.576826 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":23}
I20260927 02:44:28.577030 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.572476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43767 (local address 127.0.98.1:33107)
0927 02:44:28.572595 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.572599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.572615 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:28.572760 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.572764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.572819 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.572888 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:28.573279 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.574074 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.575543 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.576299 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.576397 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.576447 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.576778 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.576863 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:44:28.577224 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcf99af1b424dc89e961f6aec7aebaf" candidate_uuid: "26e71a93b4414e3bb237b4bbf92be938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5162b030e0504b25a9388e542c12c2a8" is_pre_election: true
I20260927 02:44:28.577539 7695 raft_consensus.cc:2454] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e71a93b4414e3bb237b4bbf92be938 in term 0.
I20260927 02:44:28.577565 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcf99af1b424dc89e961f6aec7aebaf" candidate_uuid: "26e71a93b4414e3bb237b4bbf92be938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d39bf310b3a486bae2e100f6115f8f9" is_pre_election: true
I20260927 02:44:28.577881 7422 raft_consensus.cc:2454] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e71a93b4414e3bb237b4bbf92be938 in term 0.
I20260927 02:44:28.577980 7492 leader_election.cc:302] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e71a93b4414e3bb237b4bbf92be938, 5162b030e0504b25a9388e542c12c2a8; no voters:
I20260927 02:44:28.578312 7781 raft_consensus.cc:2781] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:28.578403 7781 raft_consensus.cc:482] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:28.578447 7781 raft_consensus.cc:3037] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:28.579474 7781 raft_consensus.cc:504] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.579800 7781 leader_election.cc:288] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [CANDIDATE]: Term 1 election: Requested vote from peers 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107), 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271)
I20260927 02:44:28.580130 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcf99af1b424dc89e961f6aec7aebaf" candidate_uuid: "26e71a93b4414e3bb237b4bbf92be938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d39bf310b3a486bae2e100f6115f8f9"
I20260927 02:44:28.580188 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcf99af1b424dc89e961f6aec7aebaf" candidate_uuid: "26e71a93b4414e3bb237b4bbf92be938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5162b030e0504b25a9388e542c12c2a8"
I20260927 02:44:28.580261 7422 raft_consensus.cc:3037] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:28.580307 7695 raft_consensus.cc:3037] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:28.581194 7422 raft_consensus.cc:2454] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e71a93b4414e3bb237b4bbf92be938 in term 1.
I20260927 02:44:28.581274 7695 raft_consensus.cc:2454] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e71a93b4414e3bb237b4bbf92be938 in term 1.
I20260927 02:44:28.581462 7492 leader_election.cc:302] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e71a93b4414e3bb237b4bbf92be938, 9d39bf310b3a486bae2e100f6115f8f9; no voters:
I20260927 02:44:28.581662 7781 raft_consensus.cc:2781] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:28.581907 7781 raft_consensus.cc:686] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 1 LEADER]: Becoming Leader. State: Replica: 26e71a93b4414e3bb237b4bbf92be938, State: Running, Role: LEADER
I20260927 02:44:28.582232 7781 consensus_queue.cc:237] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } }
I20260927 02:44:28.585592 6503 catalog_manager.cc:5495] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26e71a93b4414e3bb237b4bbf92be938 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "26e71a93b4414e3bb237b4bbf92be938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } health_report { overall_health: HEALTHY } } }
I20260927 02:44:28.627653 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.623663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.0.98.2:45807)
0927 02:44:28.623815 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.623818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.623834 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:28.624218 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.624223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.624312 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.624407 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:28.624754 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.625297 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.626180 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.626407 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.626503 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.626765 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:28.627227 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:28.627323 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.627400 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.627453 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 02:44:28.633946 7422 raft_consensus.cc:1261] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 1 FOLLOWER]: Refusing update from remote peer 26e71a93b4414e3bb237b4bbf92be938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:28.633975 7695 raft_consensus.cc:1261] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 1 FOLLOWER]: Refusing update from remote peer 26e71a93b4414e3bb237b4bbf92be938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:28.634729 7783 consensus_queue.cc:1035] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:28.634912 7781 consensus_queue.cc:1035] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:28.641872 7786 mvcc.cc:205] Tried to move back new op lower bound from 7333794073117466624 to 7333794072915640320. Current Snapshot: MvccSnapshot[applied={T|T < 7333794073117466624}]
I20260927 02:44:28.651615 7788 mvcc.cc:205] Tried to move back new op lower bound from 7333794073117466624 to 7333794072915640320. Current Snapshot: MvccSnapshot[applied={T|T < 7333794073117466624 or (T in {7333794073117466624})}]
I20260927 02:44:28.653002 6504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:28.653167 6504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:28.657600 6504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab504ffef5174fef83297fcd3dcf61db: Sending DeleteTablet for 3 replicas of tablet bdcf99af1b424dc89e961f6aec7aebaf
I20260927 02:44:28.658229 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcf99af1b424dc89e961f6aec7aebaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40896
I20260927 02:44:28.658274 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcf99af1b424dc89e961f6aec7aebaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35180
I20260927 02:44:28.658676 7795 tablet_replica.cc:330] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: stopping tablet replica
I20260927 02:44:28.658707 7796 tablet_replica.cc:330] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: stopping tablet replica
I20260927 02:44:28.658999 7796 raft_consensus.cc:2229] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:28.659127 7539 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcf99af1b424dc89e961f6aec7aebaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39704
I20260927 02:44:28.659291 7796 raft_consensus.cc:2258] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:28.659551 7795 raft_consensus.cc:2229] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:28.659842 7795 raft_consensus.cc:2258] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:28.660853 7796 ts_tablet_manager.cc:1905] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:28.661360 7795 ts_tablet_manager.cc:1905] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:28.661484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 6472
I20260927 02:44:28.664288 7796 ts_tablet_manager.cc:1918] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:28.664314 7795 ts_tablet_manager.cc:1918] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:28.664390 7795 log.cc:1196] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/bdcf99af1b424dc89e961f6aec7aebaf
I20260927 02:44:28.664390 7796 log.cc:1196] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/bdcf99af1b424dc89e961f6aec7aebaf
I20260927 02:44:28.664714 7795 ts_tablet_manager.cc:1939] T bdcf99af1b424dc89e961f6aec7aebaf P 9d39bf310b3a486bae2e100f6115f8f9: Deleting consensus metadata
I20260927 02:44:28.664719 7796 ts_tablet_manager.cc:1939] T bdcf99af1b424dc89e961f6aec7aebaf P 5162b030e0504b25a9388e542c12c2a8: Deleting consensus metadata
W20260927 02:44:28.669317 7629 connection.cc:535] server connection from 127.0.0.1:35180 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 02:44:28.669588 7356 connection.cc:535] server connection from 127.0.0.1:40896 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:44:28.670992 7797 tablet_replica.cc:330] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: stopping tablet replica
I20260927 02:44:28.671283 7797 raft_consensus.cc:2229] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:28.671610 7797 raft_consensus.cc:2258] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:28.673817 7797 ts_tablet_manager.cc:1905] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:28.674216 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.660458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.0.98.61:39225)
0927 02:44:28.660573 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.660577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.665971 (+ 5394us) server_negotiation.cc:407] Connection header received
0927 02:44:28.666087 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.666090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.666134 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.666233 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.666919 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.667519 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.668429 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.668684 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.668795 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.670696 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.670699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.670702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.670732 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.672317 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.672815 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.672821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.672826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.672917 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.673233 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.673238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.673240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.673441 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.673560 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.673927 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.674059 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:44:28.676853 7797 ts_tablet_manager.cc:1918] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:28.676954 7797 log.cc:1196] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/bdcf99af1b424dc89e961f6aec7aebaf
I20260927 02:44:28.677312 7797 ts_tablet_manager.cc:1939] T bdcf99af1b424dc89e961f6aec7aebaf P 26e71a93b4414e3bb237b4bbf92be938: Deleting consensus metadata
W20260927 02:44:28.678445 7495 connection.cc:441] server connection from 127.0.0.1:39704 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39704 (request call id 1) could send its response
I20260927 02:44:28.684255 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.660172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.0.98.60:36705)
0927 02:44:28.660286 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.660292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.674520 (+ 14228us) server_negotiation.cc:407] Connection header received
0927 02:44:28.674697 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.674701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.674749 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.674803 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.675678 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.676521 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.677740 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.678198 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.678300 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.680230 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.680234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.680237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.680261 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.682052 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.682493 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.682497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.682499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.682585 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.682859 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.682862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.682864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.683070 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.683190 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.683980 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.684084 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:44:28.688843 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.679171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.0.98.60:36705)
0927 02:44:28.679286 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.679290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.680584 (+ 1294us) server_negotiation.cc:407] Connection header received
0927 02:44:28.680729 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.680732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.680773 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.680816 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.681776 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.682541 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.683708 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.684066 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.684139 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.685742 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.685746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.685748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.685777 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.687124 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.687595 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.687598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.687600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.687680 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.688027 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.688030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.688032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.688191 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.688323 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.688555 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.688663 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:44:28.689177 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:28.678819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.0.98.61:39225)
0927 02:44:28.678929 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:28.678933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:28.679042 (+ 109us) server_negotiation.cc:407] Connection header received
0927 02:44:28.679195 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:28.679198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:28.679236 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:28.679281 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:28.680140 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.680861 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.682409 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:28.682740 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:28.682810 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:28.685072 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:28.685076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:28.685078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:28.685102 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:28.686958 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.687398 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.687402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.687405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.687485 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:28.688087 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:28.688091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:28.688092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:28.688299 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:28.688392 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:28.688722 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 02:44:28.688826 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:44:29.647238 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:29.646840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:36019)
0927 02:44:29.647075 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:29.647146 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
W20260927 02:44:29.647610 7468 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42067 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:29.651432 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:29.650961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:39461)
0927 02:44:29.651222 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:29.651318 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
W20260927 02:44:29.651760 7741 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42067 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:29.653653 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:29.653294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:34639)
0927 02:44:29.653512 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:29.653580 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":70,"threads_started":1}
W20260927 02:44:29.653960 7605 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42067 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:30.157631 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab504ffef5174fef83297fcd3dcf61db)
I20260927 02:44:30.157631 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab504ffef5174fef83297fcd3dcf61db)
I20260927 02:44:30.157864 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } }
I20260927 02:44:30.157881 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } }
I20260927 02:44:30.158833 7809 leader_election.cc:288] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067), f4b216448cab4b15b3e3f75ac5005c2e (127.0.98.61:39225)
I20260927 02:44:30.158867 7810 leader_election.cc:288] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067), a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60:36705)
I20260927 02:44:30.159574 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.159109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:34176)
0927 02:44:30.159378 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.159469 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":58,"threads_started":1}
I20260927 02:44:30.159580 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.159018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:34162)
0927 02:44:30.159405 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.159480 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":73,"threads_started":1}
W20260927 02:44:30.159736 7093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
W20260927 02:44:30.159737 7276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:30.160826 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.160568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:34188)
0927 02:44:30.160694 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.160751 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 02:44:30.160897 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.160654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:34196)
0927 02:44:30.160781 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.160832 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 02:44:30.160964 7276 leader_election.cc:334] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
W20260927 02:44:30.160991 7093 leader_election.cc:334] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:30.164175 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.159504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.98.60:36705)
0927 02:44:30.159794 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:30.159800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:30.160209 (+ 409us) server_negotiation.cc:407] Connection header received
0927 02:44:30.160270 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:30.160273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:30.160328 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:30.160446 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:30.160705 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:30.161519 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:30.162929 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:30.163593 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:30.163691 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:30.163755 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:30.163935 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:44:30.164033 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":230,"thread_start_us":46,"threads_started":1}
I20260927 02:44:30.164184 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.159208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36705 (local address 127.0.0.1:45586)
0927 02:44:30.160107 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.160123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:30.160203 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:30.160458 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:30.160461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:30.160479 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:30.160578 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:30.160589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:30.161638 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:30.161641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:30.162814 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:30.162822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:30.163705 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:30.163709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:30.163788 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:30.163809 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:44:30.163911 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:44:30.163999 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260927 02:44:30.164183 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.159094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39225 (local address 127.0.0.1:39158)
0927 02:44:30.159724 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.159738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:30.159913 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:30.160334 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:30.160336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:30.160355 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:30.160465 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:30.160478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:30.161426 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:30.161431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:30.162665 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:30.162674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:30.163671 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:30.163677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:30.163770 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:30.163796 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:44:30.163911 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 02:44:30.163998 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260927 02:44:30.164186 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.159360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.0.98.61:39225)
0927 02:44:30.159556 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:30.159560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:30.159805 (+ 245us) server_negotiation.cc:407] Connection header received
0927 02:44:30.160046 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:30.160050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:30.160099 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:30.160162 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:30.160597 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:30.161313 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:30.162844 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:30.163517 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:30.163640 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:30.163709 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:30.163935 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 02:44:30.164033 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":147,"thread_start_us":57,"threads_started":1}
I20260927 02:44:30.164655 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" is_pre_election: true
I20260927 02:44:30.164738 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" is_pre_election: true
I20260927 02:44:30.164870 7299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b216448cab4b15b3e3f75ac5005c2e in term 1.
I20260927 02:44:30.164937 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d5f477b8834b3b91f2a5cdf923b67c in term 1.
I20260927 02:44:30.165263 7273 leader_election.cc:302] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4d5f477b8834b3b91f2a5cdf923b67c, f4b216448cab4b15b3e3f75ac5005c2e; no voters: ab504ffef5174fef83297fcd3dcf61db
I20260927 02:44:30.165256 7091 leader_election.cc:302] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4d5f477b8834b3b91f2a5cdf923b67c, f4b216448cab4b15b3e3f75ac5005c2e; no voters: ab504ffef5174fef83297fcd3dcf61db
I20260927 02:44:30.165537 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:44:30.165596 7810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:44:30.165616 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab504ffef5174fef83297fcd3dcf61db)
I20260927 02:44:30.165670 7809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:44:30.165658 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab504ffef5174fef83297fcd3dcf61db)
I20260927 02:44:30.165710 7810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:44:30.166641 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } }
I20260927 02:44:30.166646 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } }
I20260927 02:44:30.166972 7809 leader_election.cc:288] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 2 election: Requested vote from peers ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067), f4b216448cab4b15b3e3f75ac5005c2e (127.0.98.61:39225)
I20260927 02:44:30.167217 7810 leader_election.cc:288] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [CANDIDATE]: Term 2 election: Requested vote from peers ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067), a4d5f477b8834b3b91f2a5cdf923b67c (127.0.98.60:36705)
I20260927 02:44:30.167307 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.167080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:34212)
0927 02:44:30.167202 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.167248 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 02:44:30.167309 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f4b216448cab4b15b3e3f75ac5005c2e"
I20260927 02:44:30.167460 7115 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a4d5f477b8834b3b91f2a5cdf923b67c in current term 2: Already voted for candidate f4b216448cab4b15b3e3f75ac5005c2e in this term.
I20260927 02:44:30.167459 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.167080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:34226)
0927 02:44:30.167360 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.167406 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260927 02:44:30.167572 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c"
I20260927 02:44:30.167740 7299 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f4b216448cab4b15b3e3f75ac5005c2e in current term 2: Already voted for candidate a4d5f477b8834b3b91f2a5cdf923b67c in this term.
I20260927 02:44:30.167907 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.167679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:34232)
0927 02:44:30.167778 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.167824 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:44:30.167991 7276 leader_election.cc:334] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:30.168105 7273 leader_election.cc:302] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4d5f477b8834b3b91f2a5cdf923b67c; no voters: ab504ffef5174fef83297fcd3dcf61db, f4b216448cab4b15b3e3f75ac5005c2e
I20260927 02:44:30.168247 7809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 02:44:30.168365 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.168151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:34238)
0927 02:44:30.168252 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.168306 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:44:30.168522 7093 leader_election.cc:334] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:30.168582 7093 leader_election.cc:302] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4b216448cab4b15b3e3f75ac5005c2e; no voters: a4d5f477b8834b3b91f2a5cdf923b67c, ab504ffef5174fef83297fcd3dcf61db
I20260927 02:44:30.168699 7810 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 02:44:30.648700 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.648227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:54825)
0927 02:44:30.648516 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.648585 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":125,"threads_started":1}
I20260927 02:44:30.652772 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.652320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:37543)
0927 02:44:30.652603 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.652669 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260927 02:44:30.654831 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:30.654491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:56757)
0927 02:44:30.654701 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:30.654749 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260927 02:44:31.649863 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.649460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:35909)
0927 02:44:31.649701 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.649749 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260927 02:44:31.653779 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.653443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:49769)
0927 02:44:31.653661 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.653707 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260927 02:44:31.655774 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.655469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:59373)
0927 02:44:31.655658 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.655704 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":52,"threads_started":1}
I20260927 02:44:31.734593 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:31.734714 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } }
I20260927 02:44:31.735040 7825 leader_election.cc:288] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067), f4b216448cab4b15b3e3f75ac5005c2e (127.0.98.61:39225)
I20260927 02:44:31.735458 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" is_pre_election: true
I20260927 02:44:31.735628 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.735220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:36370)
0927 02:44:31.735460 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.735527 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
I20260927 02:44:31.735613 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d5f477b8834b3b91f2a5cdf923b67c in term 2.
I20260927 02:44:31.735985 7273 leader_election.cc:302] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4d5f477b8834b3b91f2a5cdf923b67c, f4b216448cab4b15b3e3f75ac5005c2e; no voters:
I20260927 02:44:31.736184 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 02:44:31.736263 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:31.736335 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:44:31.736454 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.736233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:36374)
0927 02:44:31.736337 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.736388 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:44:31.736577 7276 leader_election.cc:334] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:31.737385 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } }
I20260927 02:44:31.737713 7825 leader_election.cc:288] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 3 election: Requested vote from peers ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067), f4b216448cab4b15b3e3f75ac5005c2e (127.0.98.61:39225)
I20260927 02:44:31.738137 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.737868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:36388)
0927 02:44:31.738018 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.738083 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 02:44:31.738131 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f4b216448cab4b15b3e3f75ac5005c2e"
I20260927 02:44:31.738241 7115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:44:31.738796 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.738600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:36402)
0927 02:44:31.738699 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.738745 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:44:31.738925 7276 leader_election.cc:334] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
I20260927 02:44:31.739401 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d5f477b8834b3b91f2a5cdf923b67c in term 3.
I20260927 02:44:31.739675 7273 leader_election.cc:302] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4d5f477b8834b3b91f2a5cdf923b67c, f4b216448cab4b15b3e3f75ac5005c2e; no voters: ab504ffef5174fef83297fcd3dcf61db
I20260927 02:44:31.739852 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:44:31.740115 7825 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [term 3 LEADER]: Becoming Leader. State: Replica: a4d5f477b8834b3b91f2a5cdf923b67c, State: Running, Role: LEADER
I20260927 02:44:31.740375 7825 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } }
I20260927 02:44:31.741689 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d5f477b8834b3b91f2a5cdf923b67c. Latest consensus state: current_term: 3 leader_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:31.741807 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:31.742095 7830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:31.744517 7830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:31.744843 7830 catalog_manager.cc:1259] Loaded cluster ID: e59f773e4b784f129d6a5bc439c8053a
I20260927 02:44:31.744920 7830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:31.745492 7830 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:31.745817 7830 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Loaded TSK: 0
I20260927 02:44:31.746424 7830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:31.812556 7289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:31.815574 7289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:31.818506 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.818251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:36430)
0927 02:44:31.818375 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.818431 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 02:44:31.818511 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [term 3 FOLLOWER]: Refusing update from remote peer a4d5f477b8834b3b91f2a5cdf923b67c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 02:44:31.818970 7828 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 02:44:31.819243 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.819038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.0.1:36438)
0927 02:44:31.819141 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.819184 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:44:31.819353 7276 consensus_peers.cc:487] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c -> Peer ab504ffef5174fef83297fcd3dcf61db (127.0.98.62:42067): Couldn't send request to peer ab504ffef5174fef83297fcd3dcf61db. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:44:31.820412 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d5f477b8834b3b91f2a5cdf923b67c. Latest consensus state: current_term: 3 leader_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:31.820544 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:31.821662 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:31.821772 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:31.824250 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab504ffef5174fef83297fcd3dcf61db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } peers { permanent_uuid: "f4b216448cab4b15b3e3f75ac5005c2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39225 } attrs { promote: false } } peers { permanent_uuid: "a4d5f477b8834b3b91f2a5cdf923b67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36705 } attrs { promote: false } } }
I20260927 02:44:31.824340 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b216448cab4b15b3e3f75ac5005c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:31.832553 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.827968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.0.98.2:45807)
0927 02:44:31.828218 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:31.828222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:31.828237 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:44:31.828350 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:31.828353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:31.828401 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:31.828490 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:31.828741 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.829511 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.831305 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.831968 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.832067 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:31.832134 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:31.832330 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 02:44:31.832424 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260927 02:44:31.832655 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.827356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33107 (local address 127.0.0.1:41452)
0927 02:44:31.827758 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.827773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:31.827855 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:31.828329 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:31.828331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:31.828348 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:31.828482 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:31.828490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.829623 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.829626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:31.831166 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:31.831176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.832255 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.832259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:31.832350 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:31.832371 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:44:31.832465 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:44:31.832533 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":123}
I20260927 02:44:31.832819 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.827479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.0.98.1:33107)
0927 02:44:31.828066 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:31.828071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:31.828087 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:31.828145 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:31.828148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:31.828207 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:31.828591 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:31.828698 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.829503 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.831514 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.832145 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.832245 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:31.832299 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:31.832480 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:44:31.832552 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":89,"threads_started":1}
I20260927 02:44:31.833115 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.827650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45807 (local address 127.0.0.1:34912)
0927 02:44:31.828159 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.828173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:31.828260 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:31.828510 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:31.828514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:31.828535 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:31.828619 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:31.828625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.829623 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.829626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:31.831186 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:31.831192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.832086 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.832088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:31.832190 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:31.832215 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:44:31.832911 (+ 696us) client_negotiation.cc:240] Negotiation successful
0927 02:44:31.832987 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":191,"thread_start_us":80,"threads_started":1}
I20260927 02:44:31.833346 7402 tablet_service.cc:1467] Processing CreateTablet for tablet 68d49d24438b48658429f62153df247a (DEFAULT_TABLE table=test-table [id=920cf019b459486abe77a893b65a965c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:31.833457 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.828012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35271 (local address 127.0.0.1:38802)
0927 02:44:31.828716 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.828729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:31.828804 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:31.829939 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:31.829942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:31.829958 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:31.830073 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:31.830084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.830805 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.830811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:31.832068 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:31.832075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.833051 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.833054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:31.833143 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:31.833172 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 02:44:31.833251 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:44:31.833313 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":493,"thread_start_us":84,"threads_started":1}
I20260927 02:44:31.833523 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.828342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.98.3:35271)
0927 02:44:31.829739 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:31.829742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:31.829753 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:44:31.829792 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:31.829794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:31.829829 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:31.829933 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:31.830186 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.830700 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.832188 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:31.832827 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:31.832944 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:31.833000 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:31.833301 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 02:44:31.833387 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":102,"threads_started":1}
I20260927 02:44:31.833643 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d49d24438b48658429f62153df247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:31.833763 7539 tablet_service.cc:1467] Processing CreateTablet for tablet 68d49d24438b48658429f62153df247a (DEFAULT_TABLE table=test-table [id=920cf019b459486abe77a893b65a965c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:31.833964 7539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d49d24438b48658429f62153df247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:31.833956 7675 tablet_service.cc:1467] Processing CreateTablet for tablet 68d49d24438b48658429f62153df247a (DEFAULT_TABLE table=test-table [id=920cf019b459486abe77a893b65a965c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:31.834185 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d49d24438b48658429f62153df247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:31.836342 7844 tablet_bootstrap.cc:492] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9: Bootstrap starting.
I20260927 02:44:31.836334 7843 tablet_bootstrap.cc:492] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: Bootstrap starting.
I20260927 02:44:31.837249 7843 tablet_bootstrap.cc:654] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:31.837250 7844 tablet_bootstrap.cc:654] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:31.837888 7845 tablet_bootstrap.cc:492] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: Bootstrap starting.
I20260927 02:44:31.838495 7843 tablet_bootstrap.cc:492] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: No bootstrap required, opened a new log
I20260927 02:44:31.838598 7843 ts_tablet_manager.cc:1397] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:31.838904 7845 tablet_bootstrap.cc:654] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:31.839067 7844 tablet_bootstrap.cc:492] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9: No bootstrap required, opened a new log
I20260927 02:44:31.838997 7843 raft_consensus.cc:348] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:31.839133 7843 raft_consensus.cc:374] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:31.839150 7844 ts_tablet_manager.cc:1397] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:31.839196 7843 raft_consensus.cc:729] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e71a93b4414e3bb237b4bbf92be938, State: Initialized, Role: FOLLOWER
I20260927 02:44:31.839329 7843 consensus_queue.cc:260] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:31.839663 7843 ts_tablet_manager.cc:1428] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:31.839627 7844 raft_consensus.cc:348] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:31.839730 7844 raft_consensus.cc:374] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:31.839768 7844 raft_consensus.cc:729] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d39bf310b3a486bae2e100f6115f8f9, State: Initialized, Role: FOLLOWER
I20260927 02:44:31.839927 7844 consensus_queue.cc:260] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:31.840281 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.839990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:50683)
0927 02:44:31.840160 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.840214 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 02:44:31.840305 7844 ts_tablet_manager.cc:1428] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:31.841233 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.840843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:45049)
0927 02:44:31.841111 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.841166 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260927 02:44:31.841974 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.840843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:39491)
0927 02:44:31.841852 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.841902 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":944}
I20260927 02:44:31.842324 7469 heartbeater.cc:503] Master 127.0.98.60:36705 was elected leader, sending a full tablet report...
I20260927 02:44:31.842428 7845 tablet_bootstrap.cc:492] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: No bootstrap required, opened a new log
I20260927 02:44:31.842499 7845 ts_tablet_manager.cc:1397] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 02:44:31.842916 7845 raft_consensus.cc:348] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:31.843030 7845 raft_consensus.cc:374] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:31.843065 7845 raft_consensus.cc:729] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5162b030e0504b25a9388e542c12c2a8, State: Initialized, Role: FOLLOWER
I20260927 02:44:31.843178 7845 consensus_queue.cc:260] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:31.843436 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.842826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:58129)
0927 02:44:31.843297 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.843352 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
I20260927 02:44:31.843575 7845 ts_tablet_manager.cc:1428] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 02:44:31.844122 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.843862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:38493)
0927 02:44:31.843991 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.844056 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 02:44:31.844743 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:31.844509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:54239)
0927 02:44:31.844639 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:31.844686 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 02:44:31.845436 7606 heartbeater.cc:503] Master 127.0.98.60:36705 was elected leader, sending a full tablet report...
W20260927 02:44:31.932188 7744 tablet.cc:2369] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:44:31.993240 7608 tablet.cc:2369] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:32.127451 7849 raft_consensus.cc:482] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:32.127614 7849 raft_consensus.cc:504] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:32.128130 7849 leader_election.cc:288] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107), 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271)
I20260927 02:44:32.128634 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d49d24438b48658429f62153df247a" candidate_uuid: "26e71a93b4414e3bb237b4bbf92be938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d39bf310b3a486bae2e100f6115f8f9" is_pre_election: true
I20260927 02:44:32.128695 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d49d24438b48658429f62153df247a" candidate_uuid: "26e71a93b4414e3bb237b4bbf92be938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5162b030e0504b25a9388e542c12c2a8" is_pre_election: true
I20260927 02:44:32.128809 7422 raft_consensus.cc:2454] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e71a93b4414e3bb237b4bbf92be938 in term 0.
I20260927 02:44:32.128839 7695 raft_consensus.cc:2454] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e71a93b4414e3bb237b4bbf92be938 in term 0.
I20260927 02:44:32.129153 7492 leader_election.cc:302] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e71a93b4414e3bb237b4bbf92be938, 5162b030e0504b25a9388e542c12c2a8; no voters:
I20260927 02:44:32.129410 7849 raft_consensus.cc:2781] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:32.129478 7849 raft_consensus.cc:482] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:32.129537 7849 raft_consensus.cc:3037] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:32.130544 7849 raft_consensus.cc:504] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:32.130890 7849 leader_election.cc:288] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [CANDIDATE]: Term 1 election: Requested vote from peers 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107), 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271)
I20260927 02:44:32.131281 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d49d24438b48658429f62153df247a" candidate_uuid: "26e71a93b4414e3bb237b4bbf92be938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d39bf310b3a486bae2e100f6115f8f9"
I20260927 02:44:32.131361 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d49d24438b48658429f62153df247a" candidate_uuid: "26e71a93b4414e3bb237b4bbf92be938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5162b030e0504b25a9388e542c12c2a8"
I20260927 02:44:32.131424 7422 raft_consensus.cc:3037] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:32.131496 7695 raft_consensus.cc:3037] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:32.132445 7695 raft_consensus.cc:2454] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e71a93b4414e3bb237b4bbf92be938 in term 1.
I20260927 02:44:32.132485 7422 raft_consensus.cc:2454] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e71a93b4414e3bb237b4bbf92be938 in term 1.
I20260927 02:44:32.132714 7492 leader_election.cc:302] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e71a93b4414e3bb237b4bbf92be938, 5162b030e0504b25a9388e542c12c2a8; no voters:
I20260927 02:44:32.132884 7849 raft_consensus.cc:2781] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:32.133109 7849 raft_consensus.cc:686] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 1 LEADER]: Becoming Leader. State: Replica: 26e71a93b4414e3bb237b4bbf92be938, State: Running, Role: LEADER
I20260927 02:44:32.133483 7849 consensus_queue.cc:237] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } }
I20260927 02:44:32.133796 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.133542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:58205)
0927 02:44:32.133670 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:32.133733 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 02:44:32.135284 7288 catalog_manager.cc:5495] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26e71a93b4414e3bb237b4bbf92be938 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "26e71a93b4414e3bb237b4bbf92be938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e71a93b4414e3bb237b4bbf92be938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:32.193638 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.190485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.0.98.2:45807)
0927 02:44:32.190641 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:32.190646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:32.190661 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:44:32.190786 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:32.190789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:32.190846 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:32.190938 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:32.191178 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:32.191689 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:32.192615 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:32.192836 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:32.192949 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:32.193197 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:32.193266 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:32.193345 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:32.193444 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 02:44:32.193494 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:44:32.195833 7695 raft_consensus.cc:1261] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 1 FOLLOWER]: Refusing update from remote peer 26e71a93b4414e3bb237b4bbf92be938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:32.195966 7422 raft_consensus.cc:1261] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 1 FOLLOWER]: Refusing update from remote peer 26e71a93b4414e3bb237b4bbf92be938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:32.196350 7852 consensus_queue.cc:1035] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5162b030e0504b25a9388e542c12c2a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:32.196537 7849 consensus_queue.cc:1035] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d39bf310b3a486bae2e100f6115f8f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:32.197927 7742 heartbeater.cc:503] Master 127.0.98.60:36705 was elected leader, sending a full tablet report...
I20260927 02:44:32.198179 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.197656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:59225)
0927 02:44:32.198058 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:32.198116 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334}
I20260927 02:44:32.198727 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.198446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:45835)
0927 02:44:32.198608 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:32.198661 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 02:44:32.202373 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.201782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:38829)
0927 02:44:32.202253 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:32.202305 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
I20260927 02:44:32.202574 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.201944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:47927)
0927 02:44:32.202461 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:32.202509 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449}
I20260927 02:44:32.203773 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.203351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:41711)
0927 02:44:32.203642 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:32.203697 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
I20260927 02:44:32.206256 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.206026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:59483)
0927 02:44:32.206136 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:32.206185 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 02:44:32.207686 7287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:32.207870 7287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:32.211611 7287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4d5f477b8834b3b91f2a5cdf923b67c: Sending DeleteTablet for 3 replicas of tablet 68d49d24438b48658429f62153df247a
I20260927 02:44:32.212258 7539 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d49d24438b48658429f62153df247a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:32 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34912
I20260927 02:44:32.212369 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d49d24438b48658429f62153df247a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:32 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38802
I20260927 02:44:32.212594 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d49d24438b48658429f62153df247a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:32 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41452
I20260927 02:44:32.212706 7865 tablet_replica.cc:330] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: stopping tablet replica
I20260927 02:44:32.212849 7865 raft_consensus.cc:2229] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:32.212960 7866 tablet_replica.cc:330] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9: stopping tablet replica
I20260927 02:44:32.212977 7865 raft_consensus.cc:2258] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:32.213083 7866 raft_consensus.cc:2229] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:32.213209 7866 raft_consensus.cc:2258] T 68d49d24438b48658429f62153df247a P 9d39bf310b3a486bae2e100f6115f8f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:32.213250 7864 tablet_replica.cc:330] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: stopping tablet replica
I20260927 02:44:32.213475 7864 raft_consensus.cc:2229] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:32.213485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 7251
I20260927 02:44:32.214046 7864 raft_consensus.cc:2258] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:32.214591 7864 ts_tablet_manager.cc:1905] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:32.216926 7864 ts_tablet_manager.cc:1918] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:32.216984 7864 log.cc:1196] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/68d49d24438b48658429f62153df247a
I20260927 02:44:32.217195 7864 ts_tablet_manager.cc:1939] T 68d49d24438b48658429f62153df247a P 26e71a93b4414e3bb237b4bbf92be938: Deleting consensus metadata
I20260927 02:44:32.218118 7276 catalog_manager.cc:4841] TS 26e71a93b4414e3bb237b4bbf92be938 (127.0.98.2:45807): tablet 68d49d24438b48658429f62153df247a (table test-table [id=920cf019b459486abe77a893b65a965c]) successfully deleted
I20260927 02:44:32.220378 7865 ts_tablet_manager.cc:1905] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:32.223402 7865 ts_tablet_manager.cc:1918] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:32.223486 7865 log.cc:1196] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/68d49d24438b48658429f62153df247a
I20260927 02:44:32.223575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 7476
I20260927 02:44:32.223901 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:32.223596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33107 (local address 127.0.0.1:41466)
0927 02:44:32.223724 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:32.223781 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 02:44:32.224414 7273 catalog_manager.cc:4584] TS 9d39bf310b3a486bae2e100f6115f8f9 (127.0.98.1:33107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68d49d24438b48658429f62153df247a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33107: connect: Connection refused (error 111)
I20260927 02:44:32.225184 7865 ts_tablet_manager.cc:1939] T 68d49d24438b48658429f62153df247a P 5162b030e0504b25a9388e542c12c2a8: Deleting consensus metadata
I20260927 02:44:32.225917 7273 catalog_manager.cc:4841] TS 5162b030e0504b25a9388e542c12c2a8 (127.0.98.3:35271): tablet 68d49d24438b48658429f62153df247a (table test-table [id=920cf019b459486abe77a893b65a965c]) successfully deleted
I20260927 02:44:32.230090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 7612
I20260927 02:44:32.235848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 6968
I20260927 02:44:32.241539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 7078
2026-09-27T02:44:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7182 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:32 dist-test-slave-gkrl krb5kdc[7869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:32 dist-test-slave-gkrl krb5kdc[7869](info): set up 2 sockets
Sep 27 02:44:32 dist-test-slave-gkrl krb5kdc[7869](info): commencing operation
krb5kdc: starting...
W20260927 02:44:33.272519 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:33 dist-test-slave-gkrl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:44:33.389204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:33.494166 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:33.494454 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:33.494534 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:33.498451 7882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:33.498538 7882 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:44:33.498570 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:33.498612 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:33.498652 7882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:33.498689 7882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:33.503695 7882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33527
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:33.504653 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:33.505587 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:33.513695 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:33.513720 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:33.513693 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:33.514195 7882 server_base.cc:1034] running on GCE node
I20260927 02:44:33.514684 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:33.515713 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:33.516913 7882 hybrid_clock.cc:648] HybridClock initialized: now 1790477073516904 us; error 60 us; skew 500 ppm
Sep 27 02:44:33 dist-test-slave-gkrl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:33.519564 7882 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:44:33.520429 7882 webserver.cc:460] Webserver started at http://127.0.98.62:43725/ using document root <none> and password file <none>
I20260927 02:44:33.520890 7882 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:33.520946 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:33.521096 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:33.522944 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "b1ecc54309a84d36900e9b008fc70581"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "78fb3d49fd0cab06345241845acd8d7c"
server_key_iv: "861ad60f9066d3e72cd020fbf89571d1"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.523447 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1ecc54309a84d36900e9b008fc70581"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "78fb3d49fd0cab06345241845acd8d7c"
server_key_iv: "861ad60f9066d3e72cd020fbf89571d1"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.526907 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:44:33.529222 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:33.530397 7882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:33.530526 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "b1ecc54309a84d36900e9b008fc70581"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "78fb3d49fd0cab06345241845acd8d7c"
server_key_iv: "861ad60f9066d3e72cd020fbf89571d1"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.530623 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:33.545415 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:33.548059 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:33.548234 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:33.555789 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33527
I20260927 02:44:33.555806 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33527 every 8 connection(s)
I20260927 02:44:33.556730 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:33.559273 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:33.564363 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: Bootstrap starting.
I20260927 02:44:33.564667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 7882
I20260927 02:44:33.564777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:33.565117 392 external_mini_cluster.cc:1428] Setting key 52d11763d726812c1e786bae70e7a756
I20260927 02:44:33.567003 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:33.567730 7960 log.cc:824] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:33.569527 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: No bootstrap required, opened a new log
I20260927 02:44:33.573011 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ecc54309a84d36900e9b008fc70581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33527 } }
I20260927 02:44:33.573248 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:33.573329 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ecc54309a84d36900e9b008fc70581, State: Initialized, Role: FOLLOWER
Sep 27 02:44:33 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:33.573778 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ecc54309a84d36900e9b008fc70581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33527 } }
I20260927 02:44:33.573922 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:33.574007 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:33.574110 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:33.575232 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ecc54309a84d36900e9b008fc70581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33527 } }
I20260927 02:44:33.575598 7960 leader_election.cc:302] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ecc54309a84d36900e9b008fc70581; no voters:
I20260927 02:44:33.575915 7960 leader_election.cc:288] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:33.576195 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:33.576400 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [term 1 LEADER]: Becoming Leader. State: Replica: b1ecc54309a84d36900e9b008fc70581, State: Running, Role: LEADER
I20260927 02:44:33.576705 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ecc54309a84d36900e9b008fc70581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33527 } }
I20260927 02:44:33.577171 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:33.578423 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1ecc54309a84d36900e9b008fc70581. Latest consensus state: current_term: 1 leader_uuid: "b1ecc54309a84d36900e9b008fc70581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ecc54309a84d36900e9b008fc70581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33527 } } }
I20260927 02:44:33.578550 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:33.578915 7974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:33.578944 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1ecc54309a84d36900e9b008fc70581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ecc54309a84d36900e9b008fc70581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33527 } } }
I20260927 02:44:33.579012 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:33.579979 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:33.566405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.0.98.62:33527)
0927 02:44:33.566790 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:33.566806 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:33.566839 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:44:33.567495 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:33.567523 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:33.567864 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:33.568277 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:33.569038 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.570078 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.570928 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.571381 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.571543 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:33.574122 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:33.574130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:33.574146 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:33.574178 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:33.576202 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:33.576687 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:33.576693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:33.576698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:33.576853 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:33.577166 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:33.577169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:33.577170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:33.577668 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:33.577854 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:33.578606 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 02:44:33.578926 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
I20260927 02:44:33.581947 7974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:33.586915 7974 catalog_manager.cc:1347] Generated new cluster ID: 560cea901cf1457398f53652ad0a6135
I20260927 02:44:33.586984 7974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:33.610509 7974 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:33.611765 7974 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:33.618954 7974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: Generated new TSK 0
I20260927 02:44:33.619688 7974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:44:33.647603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33527
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:33.756639 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:33.756918 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:33.757009 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:33.760960 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:33.761031 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:33.761133 7986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:33.761209 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:33.766690 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:33.767647 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:33.768635 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:33.777865 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:33.777926 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:33.777884 7986 server_base.cc:1034] running on GCE node
W20260927 02:44:33.778056 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:33.778458 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:33.779138 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:33.780319 7986 hybrid_clock.cc:648] HybridClock initialized: now 1790477073780292 us; error 37 us; skew 500 ppm
Sep 27 02:44:33 dist-test-slave-gkrl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:33.782904 7986 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:44:33.783692 7986 webserver.cc:460] Webserver started at http://127.0.98.1:35951/ using document root <none> and password file <none>
I20260927 02:44:33.784265 7986 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:33.784377 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:33.784583 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:33.786442 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "5330fab27c8b4c3b91adea7dd874acdd"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "19f20180249010d502836c298f542cae"
server_key_iv: "fdceed887041ab115a8d95717212f150"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.786962 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5330fab27c8b4c3b91adea7dd874acdd"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "19f20180249010d502836c298f542cae"
server_key_iv: "fdceed887041ab115a8d95717212f150"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.790490 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:44:33.792725 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:33.793915 7986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:33.794059 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "5330fab27c8b4c3b91adea7dd874acdd"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "19f20180249010d502836c298f542cae"
server_key_iv: "fdceed887041ab115a8d95717212f150"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.794178 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:33.805270 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:33.807950 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:33.808132 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:33.809386 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:33.809456 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:33.809525 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:33.809568 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:44:33 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:33.820079 7986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40537
I20260927 02:44:33.820894 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40537 every 8 connection(s)
I20260927 02:44:33.821131 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:33.822801 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:33.810325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41153 (local address 127.0.98.62:33527)
0927 02:44:33.810473 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:33.810478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:33.811204 (+ 726us) server_negotiation.cc:407] Connection header received
0927 02:44:33.812270 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:33.812277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:33.812344 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:33.812449 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:33.813619 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.814151 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.815335 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.815619 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.815732 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:33.818880 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:33.818887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:33.818893 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:33.818925 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:33.820685 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:33.821289 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:33.821292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:33.821293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:33.821366 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:33.821756 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:33.821759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:33.821761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:33.821991 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:33.822070 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:33.822582 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 02:44:33.822684 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:33.823503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 7986
I20260927 02:44:33.823637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:33.823644 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:33.810628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33527 (local address 127.0.98.1:41153)
0927 02:44:33.811044 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:33.811090 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:33.812045 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:33.812646 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:33.812653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:33.812946 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:33.813441 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:33.813480 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.814273 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.814277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:33.815204 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:33.815213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.815765 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.815769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:33.815999 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:33.816616 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:33.816631 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:33.818720 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:33.820791 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:33.820796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:33.820805 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:33.821149 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:33.821463 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:33.821466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:33.821468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:33.821633 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:33.822097 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:33.822102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:33.822298 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 02:44:33.822448 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 02:44:33.822659 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":125,"threads_started":1}
I20260927 02:44:33.823987 392 external_mini_cluster.cc:1428] Setting key 33d82baa0eba3aff28a94603a57e0684
I20260927 02:44:33.825109 8117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33527
I20260927 02:44:33.825258 8117 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:33.825707 8117 heartbeater.cc:511] Master 127.0.98.62:33527 requested a full tablet report, sending...
I20260927 02:44:33.827737 7914 ts_manager.cc:194] Registered new tserver with Master: 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537)
I20260927 02:44:33.829121 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41153
I20260927 02:44:33.835014 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:33.827069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35625 (local address 127.0.98.62:33527)
0927 02:44:33.827972 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:33.827976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:33.827989 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:33.828022 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:33.828025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:33.828061 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:33.828139 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:33.828678 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.829366 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.830461 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.830684 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.830789 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:33.831910 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:33.831914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:33.831917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:33.831939 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:33.833450 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:33.833856 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:33.833860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:33.833861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:33.833940 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:33.834205 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:33.834209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:33.834211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:33.834428 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:33.834519 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:33.834766 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:44:33.834866 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831}
I20260927 02:44:33.835024 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:33.826951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33527 (local address 127.0.98.1:35625)
0927 02:44:33.827226 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:33.827242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:33.827322 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:33.828180 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:33.828185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:33.828401 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:33.828548 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:33.828558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.829525 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.829530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:33.830344 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:33.830353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:33.830790 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:33.830793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:33.830880 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:33.831360 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:33.831380 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:33.831759 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:33.833549 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:33.833553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:33.833556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:33.833743 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:33.834027 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:33.834032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:33.834033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:33.834118 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:33.834509 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:33.834512 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:33.834604 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 02:44:33.834707 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 02:44:33.834834 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:44:33.851475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33527
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:33.957616 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:33.957878 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:33.957962 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:33.961973 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:33.962049 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:33.962118 8126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:33.962198 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:33.967204 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:33.968241 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:33.969197 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:33.978276 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:33.978263 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:33.978279 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:33.978514 8126 server_base.cc:1034] running on GCE node
I20260927 02:44:33.978870 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:33.979554 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:33.980729 8126 hybrid_clock.cc:648] HybridClock initialized: now 1790477073980719 us; error 33 us; skew 500 ppm
Sep 27 02:44:33 dist-test-slave-gkrl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:33.983421 8126 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:44:33.984237 8126 webserver.cc:460] Webserver started at http://127.0.98.2:46373/ using document root <none> and password file <none>
I20260927 02:44:33.984726 8126 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:33.984782 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:33.984938 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:33.986860 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "d23bae3431483995a45f3365e25fc3bd"
server_key_iv: "8fd8cf90c1816381737059ce6aef3bdd"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.987344 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "d23bae3431483995a45f3365e25fc3bd"
server_key_iv: "8fd8cf90c1816381737059ce6aef3bdd"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.990854 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:44:33.993165 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:33.994405 8126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:33.994494 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04"
format_stamp: "Formatted at 2026-09-27 02:44:33 on dist-test-slave-gkrl"
server_key: "d23bae3431483995a45f3365e25fc3bd"
server_key_iv: "8fd8cf90c1816381737059ce6aef3bdd"
server_key_version: "encryptionkey@0"
I20260927 02:44:33.994575 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:34.016958 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:34.020309 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:34.020524 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:34.021965 8126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:34.022035 8126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:34.022099 8126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:34.022130 8126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:44:34 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:34.033910 8126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42639
I20260927 02:44:34.034683 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42639 every 8 connection(s)
I20260927 02:44:34.034834 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:34.036726 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.022740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33317 (local address 127.0.98.62:33527)
0927 02:44:34.022883 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.022888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.023725 (+ 837us) server_negotiation.cc:407] Connection header received
0927 02:44:34.024918 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.024922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.024985 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.025072 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:34.026282 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.026813 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.028146 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.028379 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.028496 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.032811 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:34.032817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:34.032821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:34.032851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:34.034547 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.035141 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.035144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.035145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.035219 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.035646 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.035651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.035653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.035931 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:34.036027 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.036508 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.036613 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:44:34.037180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 8126
I20260927 02:44:34.037286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:34.037542 392 external_mini_cluster.cc:1428] Setting key f811841e1b6213bf8e75194fc875e997
I20260927 02:44:34.037539 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.023136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33527 (local address 127.0.98.2:33317)
0927 02:44:34.023562 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.023610 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.024651 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.025267 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.025281 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.025584 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:34.026149 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.026169 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.026994 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.026997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.028031 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.028040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.028528 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.028533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.028717 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.029276 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:34.029294 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:34.032609 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:34.034665 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.034670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.034684 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.035048 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.035333 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.035339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.035342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.035499 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.036055 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:34.036061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:34.036277 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.036424 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.036658 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":123,"threads_started":1}
I20260927 02:44:34.038832 8257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33527
I20260927 02:44:34.038960 8257 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:34.039358 8257 heartbeater.cc:511] Master 127.0.98.62:33527 requested a full tablet report, sending...
I20260927 02:44:34.041131 7914 ts_manager.cc:194] Registered new tserver with Master: ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639)
I20260927 02:44:34.041908 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33317
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:34.048419 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.041284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42731 (local address 127.0.98.62:33527)
0927 02:44:34.041381 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.041385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.041611 (+ 226us) server_negotiation.cc:407] Connection header received
0927 02:44:34.041760 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.041763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.041799 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.041843 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:34.042423 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.042946 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.043851 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.044117 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.044224 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.045270 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:34.045274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:34.045276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:34.045303 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:34.046763 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.047199 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.047204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.047205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.047282 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.047565 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.047569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.047571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.047780 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:34.047929 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.048160 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.048260 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260927 02:44:34.048451 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.041230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33527 (local address 127.0.98.2:42731)
0927 02:44:34.041536 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.041550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.041669 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.041971 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.041977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.042176 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:34.042298 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.042306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.043070 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.043072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.043735 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.043744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.044225 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.044228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.044317 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.044803 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:34.044817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:34.045135 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:34.046884 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.046887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.046892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.047097 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.047372 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.047376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.047378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.047469 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.047900 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:34.047903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:34.047990 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.048110 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.048250 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:44:34.065035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33527
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:34.173228 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:34.173509 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:34.173588 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:34.177636 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:34.177709 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:34.177775 8266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:44:34.177850 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:34.182979 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:34.183962 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:34.184943 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:34.194249 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:34.194314 8266 server_base.cc:1034] running on GCE node
W20260927 02:44:34.194336 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:34.194237 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:34.194938 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:34.195521 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:34.196682 8266 hybrid_clock.cc:648] HybridClock initialized: now 1790477074196652 us; error 41 us; skew 500 ppm
Sep 27 02:44:34 dist-test-slave-gkrl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477074, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:34.199508 8266 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:44:34.200356 8266 webserver.cc:460] Webserver started at http://127.0.98.3:40021/ using document root <none> and password file <none>
I20260927 02:44:34.200922 8266 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:34.200984 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:34.201207 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:34.203137 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "9407d8f24453477abb5933a3964bcfd6"
format_stamp: "Formatted at 2026-09-27 02:44:34 on dist-test-slave-gkrl"
server_key: "47c0e215b5f00ab6b6cb4653205b15cc"
server_key_iv: "1c3d95d96bcf0f60b1e505258e9a000f"
server_key_version: "encryptionkey@0"
I20260927 02:44:34.203686 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9407d8f24453477abb5933a3964bcfd6"
format_stamp: "Formatted at 2026-09-27 02:44:34 on dist-test-slave-gkrl"
server_key: "47c0e215b5f00ab6b6cb4653205b15cc"
server_key_iv: "1c3d95d96bcf0f60b1e505258e9a000f"
server_key_version: "encryptionkey@0"
I20260927 02:44:34.207149 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:34.209450 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:34.210572 8266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:34.210700 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "9407d8f24453477abb5933a3964bcfd6"
format_stamp: "Formatted at 2026-09-27 02:44:34 on dist-test-slave-gkrl"
server_key: "47c0e215b5f00ab6b6cb4653205b15cc"
server_key_iv: "1c3d95d96bcf0f60b1e505258e9a000f"
server_key_version: "encryptionkey@0"
I20260927 02:44:34.210803 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:34.222784 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:34.225543 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:34.225730 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:34.227066 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:34.227142 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:34.227202 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:34.227252 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:44:34 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477074, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:34.238159 8266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37329
I20260927 02:44:34.238243 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37329 every 8 connection(s)
I20260927 02:44:34.239027 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:34.240742 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.228144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38469 (local address 127.0.98.62:33527)
0927 02:44:34.228269 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.228277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.229260 (+ 983us) server_negotiation.cc:407] Connection header received
0927 02:44:34.230251 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.230254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.230322 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.230405 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:34.231604 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.232259 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.233458 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.233803 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.233933 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.237075 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:34.237078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:34.237081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:34.237106 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:34.238657 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.239144 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.239148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.239150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.239229 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.239595 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.239599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.239600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.239806 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:34.239931 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.240405 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.240534 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:44:34.240856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 8266
I20260927 02:44:34.241009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:34.241397 392 external_mini_cluster.cc:1428] Setting key 6deac83f9fda209c9ce16c790a713fe6
I20260927 02:44:34.241678 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.228639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33527 (local address 127.0.98.3:38469)
0927 02:44:34.229105 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.229143 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.230052 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.230613 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.230624 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.230939 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:34.231460 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.231476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.232386 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.232390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.233348 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.233361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.233972 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.233980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.234210 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.234849 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:34.234867 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:34.236920 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:34.238768 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.238775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.238788 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.239055 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.239322 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.239325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.239327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.239487 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.239931 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:34.239937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:34.240171 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.240325 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.240667 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":142,"threads_started":1}
I20260927 02:44:34.243080 8397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33527
I20260927 02:44:34.243247 8397 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:34.243796 8397 heartbeater.cc:511] Master 127.0.98.62:33527 requested a full tablet report, sending...
I20260927 02:44:34.244848 7914 ts_manager.cc:194] Registered new tserver with Master: 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:34.245611 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38469
I20260927 02:44:34.251820 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.245122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55517 (local address 127.0.98.62:33527)
0927 02:44:34.245258 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.245263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.245491 (+ 228us) server_negotiation.cc:407] Connection header received
0927 02:44:34.245569 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.245571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.245606 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.245674 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:34.246427 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.246947 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.247995 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.248202 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.248285 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.249157 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:34.249160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:34.249161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:34.249181 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:34.250534 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.250905 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.250940 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.250941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.251008 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.251245 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.251247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.251253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.251394 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:34.251466 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.251651 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.251725 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:44:34.251832 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.245078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33527 (local address 127.0.98.3:55517)
0927 02:44:34.245386 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.245400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.245492 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.245716 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.245719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.246014 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:34.246115 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.246121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.247078 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.247082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.247838 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.247846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.248297 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.248309 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.248373 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.248760 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:34.248771 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:34.249043 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:34.250634 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.250637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.250638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.250821 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.251084 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.251086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.251088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.251158 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.251472 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:34.251474 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:34.251545 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.251622 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.251713 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":131,"threads_started":1}
I20260927 02:44:34.255286 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:44:34 dist-test-slave-gkrl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477074, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:44:34 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:34 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:44:34.278486 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:34.272390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.0.98.62:33527)
0927 02:44:34.272525 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.272528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.272539 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:44:34.272573 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.272576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.272610 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.272697 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:34.273380 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.273899 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.274745 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.274967 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.275075 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.278321 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40208: BlockingRecv error: recv got EOF from 127.0.0.1:40208 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 02:44:34 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477074, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:34.287679 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.279870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.0.98.62:33527)
0927 02:44:34.280029 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.280033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.280127 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:44:34.280216 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.280217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.280250 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.280314 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:34.280954 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.281435 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.282189 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.282381 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.282458 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.284721 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:34.284724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:34.284725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:34.284746 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:34.286221 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.286651 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.286653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.286655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.286730 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.286988 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.286990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.286991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.287157 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:34.287271 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.287478 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.287571 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 02:44:34.290354 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:34.293413 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:34.308687 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.302000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40537 (local address 127.0.0.1:39162)
0927 02:44:34.303271 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.303284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.303458 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.304191 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.304197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.304217 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:34.304344 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.304351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.305742 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.305745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.307291 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.307298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.308161 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.308164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.308279 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.308319 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.308426 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.308553 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":83,"spinlock_wait_cycles":23040,"thread_start_us":103,"threads_started":1}
I20260927 02:44:34.308697 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.302206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.0.98.1:40537)
0927 02:44:34.302443 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.302460 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.303368 (+ 908us) server_negotiation.cc:407] Connection header received
0927 02:44:34.303626 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.303632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.303848 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.304047 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:34.304464 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.305536 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.307413 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.308041 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.308162 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.308325 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.308455 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.308544 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260927 02:44:34.309163 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.302568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.0.98.2:42639)
0927 02:44:34.302843 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.302855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.303267 (+ 412us) server_negotiation.cc:407] Connection header received
0927 02:44:34.303536 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.303544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.303770 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.304207 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:34.304326 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.305479 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.307291 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.308162 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.308257 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.308416 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.308932 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.309020 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
I20260927 02:44:34.309357 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.302735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37329 (local address 127.0.0.1:34048)
0927 02:44:34.303606 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.303617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.303699 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.304220 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.304223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.304244 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:34.304348 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.304353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.306767 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.306773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.307898 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.307909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.308949 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.308952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.309037 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.309058 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.309115 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.309205 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":72,"spinlock_wait_cycles":1408,"thread_start_us":84,"threads_started":1}
I20260927 02:44:34.309386 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.302892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.0.98.3:37329)
0927 02:44:34.303220 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.303234 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.303710 (+ 476us) server_negotiation.cc:407] Connection header received
0927 02:44:34.303782 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.303786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.303986 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.304104 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:34.305664 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.306655 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.308036 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.308826 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.308948 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.309084 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.309135 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.309230 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":130,"threads_started":1}
I20260927 02:44:34.309652 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.301968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42639 (local address 127.0.0.1:46872)
0927 02:44:34.303124 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.303172 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.303385 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.303959 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.303966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.303990 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:34.304135 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.304143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.305619 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.305623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.307162 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.307172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.308474 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.308478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.308570 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.308591 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.308662 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.308731 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":66,"thread_start_us":116,"threads_started":1}
I20260927 02:44:34.312176 8035 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8885d9de134a08bcdefedd316b7178 (DEFAULT_TABLE table=test-table [id=99771949544f47069e2cb572f1ae65e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:34.312534 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8885d9de134a08bcdefedd316b7178 (DEFAULT_TABLE table=test-table [id=99771949544f47069e2cb572f1ae65e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:34.313285 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8885d9de134a08bcdefedd316b7178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.313442 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8885d9de134a08bcdefedd316b7178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.313612 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8885d9de134a08bcdefedd316b7178 (DEFAULT_TABLE table=test-table [id=99771949544f47069e2cb572f1ae65e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:34.314524 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8885d9de134a08bcdefedd316b7178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.320727 8425 tablet_bootstrap.cc:492] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: Bootstrap starting.
I20260927 02:44:34.321663 8426 tablet_bootstrap.cc:492] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Bootstrap starting.
I20260927 02:44:34.322789 8425 tablet_bootstrap.cc:654] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.323361 8426 tablet_bootstrap.cc:654] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.323551 8425 log.cc:824] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:34.324191 8426 log.cc:824] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:34.324220 8427 tablet_bootstrap.cc:492] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: Bootstrap starting.
I20260927 02:44:34.325651 8425 tablet_bootstrap.cc:492] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: No bootstrap required, opened a new log
I20260927 02:44:34.325882 8425 ts_tablet_manager.cc:1397] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:44:34.326128 8427 tablet_bootstrap.cc:654] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.326752 8427 log.cc:824] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:34.328440 8426 tablet_bootstrap.cc:492] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: No bootstrap required, opened a new log
I20260927 02:44:34.328611 8427 tablet_bootstrap.cc:492] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: No bootstrap required, opened a new log
I20260927 02:44:34.328640 8426 ts_tablet_manager.cc:1397] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260927 02:44:34.328783 8427 ts_tablet_manager.cc:1397] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:44:34.329720 8425 raft_consensus.cc:348] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.329996 8425 raft_consensus.cc:374] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.330050 8425 raft_consensus.cc:729] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9407d8f24453477abb5933a3964bcfd6, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.330516 8425 consensus_queue.cc:260] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.331465 8425 ts_tablet_manager.cc:1428] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:44:34.331838 8397 heartbeater.cc:503] Master 127.0.98.62:33527 was elected leader, sending a full tablet report...
I20260927 02:44:34.332101 8426 raft_consensus.cc:348] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.332346 8426 raft_consensus.cc:374] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.332401 8426 raft_consensus.cc:729] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2c26b1fc634ca5a12fd8e9c5e70e04, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.332409 8427 raft_consensus.cc:348] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.332660 8427 raft_consensus.cc:374] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.332721 8427 raft_consensus.cc:729] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5330fab27c8b4c3b91adea7dd874acdd, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.332785 8426 consensus_queue.cc:260] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.333145 8427 consensus_queue.cc:260] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.333398 8257 heartbeater.cc:503] Master 127.0.98.62:33527 was elected leader, sending a full tablet report...
I20260927 02:44:34.333738 8426 ts_tablet_manager.cc:1428] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 02:44:34.334664 8117 heartbeater.cc:503] Master 127.0.98.62:33527 was elected leader, sending a full tablet report...
I20260927 02:44:34.335024 8427 ts_tablet_manager.cc:1428] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:44:34.432915 8433 raft_consensus.cc:482] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:34.433130 8433 raft_consensus.cc:504] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.434369 8433 leader_election.cc:288] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639), 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:34.438697 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.435054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38833 (local address 127.0.98.3:37329)
0927 02:44:34.435176 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.435182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.435372 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:44:34.435458 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.435460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.435519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.435597 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:34.435824 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.436583 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.437698 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.438100 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.438180 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.438264 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.438475 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.438566 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:44:34.438786 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.434703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41317 (local address 127.0.98.2:42639)
0927 02:44:34.434848 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.434855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.434915 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:44:34.435093 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.435097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.435159 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.435243 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:34.435560 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.436473 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.437723 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.438134 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.438263 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.438325 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.438582 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.438664 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:44:34.438751 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.434922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37329 (local address 127.0.98.1:38833)
0927 02:44:34.435273 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.435285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.435372 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.435620 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.435622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.435639 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:34.435721 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.435726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.436678 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.436680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.437569 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.437578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.438231 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.438235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.438327 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.438355 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.438475 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.438573 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":5,"thread_start_us":55,"threads_started":1}
I20260927 02:44:34.438771 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.434534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42639 (local address 127.0.98.1:41317)
0927 02:44:34.434817 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.434831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.434963 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.435273 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.435278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.435304 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:34.435446 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.435455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.436614 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.436617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.437573 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.437584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.438280 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.438286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.438380 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.438434 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.438546 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.438611 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":89,"thread_start_us":107,"threads_started":1}
I20260927 02:44:34.439441 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8885d9de134a08bcdefedd316b7178" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" is_pre_election: true
I20260927 02:44:34.439445 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8885d9de134a08bcdefedd316b7178" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9407d8f24453477abb5933a3964bcfd6" is_pre_election: true
I20260927 02:44:34.439730 8351 raft_consensus.cc:2454] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 0.
I20260927 02:44:34.439730 8211 raft_consensus.cc:2454] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 0.
I20260927 02:44:34.440208 8003 leader_election.cc:302] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5330fab27c8b4c3b91adea7dd874acdd, 9407d8f24453477abb5933a3964bcfd6; no voters:
I20260927 02:44:34.440430 8433 raft_consensus.cc:2781] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:34.440503 8433 raft_consensus.cc:482] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:34.440533 8433 raft_consensus.cc:3037] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.441569 8433 raft_consensus.cc:504] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.441879 8433 leader_election.cc:288] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 election: Requested vote from peers ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639), 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:34.442242 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8885d9de134a08bcdefedd316b7178" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04"
I20260927 02:44:34.442322 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8885d9de134a08bcdefedd316b7178" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9407d8f24453477abb5933a3964bcfd6"
I20260927 02:44:34.442374 8211 raft_consensus.cc:3037] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.442452 8351 raft_consensus.cc:3037] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.443483 8211 raft_consensus.cc:2454] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 1.
I20260927 02:44:34.443547 8351 raft_consensus.cc:2454] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 1.
I20260927 02:44:34.443763 8003 leader_election.cc:302] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5330fab27c8b4c3b91adea7dd874acdd, ef2c26b1fc634ca5a12fd8e9c5e70e04; no voters:
I20260927 02:44:34.443951 8433 raft_consensus.cc:2781] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:34.444164 8433 raft_consensus.cc:686] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 LEADER]: Becoming Leader. State: Replica: 5330fab27c8b4c3b91adea7dd874acdd, State: Running, Role: LEADER
I20260927 02:44:34.444522 8433 consensus_queue.cc:237] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.447580 7913 catalog_manager.cc:5495] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5330fab27c8b4c3b91adea7dd874acdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:34.465340 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.461740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.0.98.1:40537)
0927 02:44:34.461873 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.461878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.461892 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:34.462100 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.462103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.462155 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.462264 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:34.462487 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.462993 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.463839 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.464125 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.464236 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.464524 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:34.464946 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:34.465072 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.465145 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.465198 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:44:34.470590 8351 raft_consensus.cc:1261] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Refusing update from remote peer 5330fab27c8b4c3b91adea7dd874acdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:34.470639 8211 raft_consensus.cc:1261] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Refusing update from remote peer 5330fab27c8b4c3b91adea7dd874acdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:34.471292 8436 consensus_queue.cc:1035] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:34.471493 8433 consensus_queue.cc:1035] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:34.479194 8440 mvcc.cc:205] Tried to move back new op lower bound from 7333794097025548288 to 7333794096927338496. Current Snapshot: MvccSnapshot[applied={T|T < 7333794097025548288}]
I20260927 02:44:34.480340 8441 mvcc.cc:205] Tried to move back new op lower bound from 7333794097025548288 to 7333794096927338496. Current Snapshot: MvccSnapshot[applied={T|T < 7333794097025548288}]
I20260927 02:44:34.486167 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:34.486321 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:34.488462 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: Sending DeleteTablet for 3 replicas of tablet 7c8885d9de134a08bcdefedd316b7178
I20260927 02:44:34.489192 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8885d9de134a08bcdefedd316b7178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34048
I20260927 02:44:34.489231 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8885d9de134a08bcdefedd316b7178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39162
I20260927 02:44:34.489732 8450 tablet_replica.cc:330] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: stopping tablet replica
I20260927 02:44:34.489719 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8885d9de134a08bcdefedd316b7178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46872
I20260927 02:44:34.489714 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:44:34.490005 8450 raft_consensus.cc:2229] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:34.490278 8449 tablet_replica.cc:330] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: stopping tablet replica
I20260927 02:44:34.490325 8450 raft_consensus.cc:2258] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:34.490483 8451 tablet_replica.cc:330] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: stopping tablet replica
W20260927 02:44:34.490635 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:34.490897 8451 raft_consensus.cc:2229] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:34.491240 8451 raft_consensus.cc:2258] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:34.491669 8450 ts_tablet_manager.cc:1905] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:34.492678 8451 ts_tablet_manager.cc:1905] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:34.494499 8450 ts_tablet_manager.cc:1918] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:34.494598 8450 log.cc:1196] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/7c8885d9de134a08bcdefedd316b7178
I20260927 02:44:34.495098 8035 tablet_service.cc:1467] Processing CreateTablet for tablet e83d0332c8f6486a937217401b1d5d53 (DEFAULT_TABLE table=test-table [id=849203b6d72e4016ad153145326f7404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:34.495327 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83d0332c8f6486a937217401b1d5d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.495374 8191 tablet_service.cc:1467] Processing CreateTablet for tablet e83d0332c8f6486a937217401b1d5d53 (DEFAULT_TABLE table=test-table [id=849203b6d72e4016ad153145326f7404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:34.495584 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83d0332c8f6486a937217401b1d5d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.496166 8449 raft_consensus.cc:2229] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:34.496425 8449 raft_consensus.cc:2258] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:34.497686 8426 tablet_bootstrap.cc:492] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Bootstrap starting.
I20260927 02:44:34.497717 8427 tablet_bootstrap.cc:492] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: Bootstrap starting.
I20260927 02:44:34.497886 8331 tablet_service.cc:1467] Processing CreateTablet for tablet e83d0332c8f6486a937217401b1d5d53 (DEFAULT_TABLE table=test-table [id=849203b6d72e4016ad153145326f7404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:34.498132 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83d0332c8f6486a937217401b1d5d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.498627 8427 tablet_bootstrap.cc:654] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.498693 8426 tablet_bootstrap.cc:654] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.500108 8450 ts_tablet_manager.cc:1939] T 7c8885d9de134a08bcdefedd316b7178 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting consensus metadata
I20260927 02:44:34.500569 8449 ts_tablet_manager.cc:1905] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:34.501148 7898 catalog_manager.cc:4841] TS 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537): tablet 7c8885d9de134a08bcdefedd316b7178 (table test-table [id=99771949544f47069e2cb572f1ae65e3]) successfully deleted
I20260927 02:44:34.501488 8425 tablet_bootstrap.cc:492] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: Bootstrap starting.
I20260927 02:44:34.502436 8425 tablet_bootstrap.cc:654] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.502745 8427 tablet_bootstrap.cc:492] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: No bootstrap required, opened a new log
I20260927 02:44:34.502815 8427 ts_tablet_manager.cc:1397] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 02:44:34.502806 8449 ts_tablet_manager.cc:1918] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:34.502892 8449 log.cc:1196] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/7c8885d9de134a08bcdefedd316b7178
I20260927 02:44:34.503165 8449 ts_tablet_manager.cc:1939] T 7c8885d9de134a08bcdefedd316b7178 P 9407d8f24453477abb5933a3964bcfd6: Deleting consensus metadata
I20260927 02:44:34.503269 8427 raft_consensus.cc:348] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.503392 8427 raft_consensus.cc:374] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.503438 8427 raft_consensus.cc:729] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5330fab27c8b4c3b91adea7dd874acdd, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.503564 8427 consensus_queue.cc:260] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.503819 8427 ts_tablet_manager.cc:1428] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:34.504391 7899 catalog_manager.cc:4841] TS 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329): tablet 7c8885d9de134a08bcdefedd316b7178 (table test-table [id=99771949544f47069e2cb572f1ae65e3]) successfully deleted
I20260927 02:44:34.504690 8426 tablet_bootstrap.cc:492] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: No bootstrap required, opened a new log
I20260927 02:44:34.504776 8426 ts_tablet_manager.cc:1397] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260927 02:44:34.505178 8426 raft_consensus.cc:348] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.505286 8426 raft_consensus.cc:374] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.505328 8426 raft_consensus.cc:729] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2c26b1fc634ca5a12fd8e9c5e70e04, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.505445 8426 consensus_queue.cc:260] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.505686 8426 ts_tablet_manager.cc:1428] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:34.506664 8451 ts_tablet_manager.cc:1918] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:34.506772 8451 log.cc:1196] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/7c8885d9de134a08bcdefedd316b7178
I20260927 02:44:34.507126 8451 ts_tablet_manager.cc:1939] T 7c8885d9de134a08bcdefedd316b7178 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting consensus metadata
I20260927 02:44:34.508242 7899 catalog_manager.cc:4841] TS ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639): tablet 7c8885d9de134a08bcdefedd316b7178 (table test-table [id=99771949544f47069e2cb572f1ae65e3]) successfully deleted
I20260927 02:44:34.508836 8425 tablet_bootstrap.cc:492] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: No bootstrap required, opened a new log
I20260927 02:44:34.508930 8425 ts_tablet_manager.cc:1397] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 02:44:34.509320 8425 raft_consensus.cc:348] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.509424 8425 raft_consensus.cc:374] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.509456 8425 raft_consensus.cc:729] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9407d8f24453477abb5933a3964bcfd6, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.509575 8425 consensus_queue.cc:260] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.509800 8425 ts_tablet_manager.cc:1428] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260927 02:44:34.536152 8258 tablet.cc:2369] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:44:34.572419 8118 tablet.cc:2369] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:34.648192 8444 raft_consensus.cc:482] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:34.648357 8444 raft_consensus.cc:504] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.648856 8444 leader_election.cc:288] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639), 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:34.649320 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83d0332c8f6486a937217401b1d5d53" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" is_pre_election: true
I20260927 02:44:34.649418 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83d0332c8f6486a937217401b1d5d53" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9407d8f24453477abb5933a3964bcfd6" is_pre_election: true
I20260927 02:44:34.649480 8211 raft_consensus.cc:2454] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 0.
I20260927 02:44:34.649557 8351 raft_consensus.cc:2454] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 0.
I20260927 02:44:34.649814 8003 leader_election.cc:302] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5330fab27c8b4c3b91adea7dd874acdd, 9407d8f24453477abb5933a3964bcfd6; no voters:
I20260927 02:44:34.649991 8444 raft_consensus.cc:2781] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:34.650058 8444 raft_consensus.cc:482] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:34.650110 8444 raft_consensus.cc:3037] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.651080 8444 raft_consensus.cc:504] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.651400 8444 leader_election.cc:288] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 election: Requested vote from peers ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639), 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:34.651750 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83d0332c8f6486a937217401b1d5d53" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04"
I20260927 02:44:34.651831 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83d0332c8f6486a937217401b1d5d53" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9407d8f24453477abb5933a3964bcfd6"
I20260927 02:44:34.651906 8211 raft_consensus.cc:3037] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.651961 8351 raft_consensus.cc:3037] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.653021 8211 raft_consensus.cc:2454] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 1.
I20260927 02:44:34.653016 8351 raft_consensus.cc:2454] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 1.
I20260927 02:44:34.653293 8003 leader_election.cc:302] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5330fab27c8b4c3b91adea7dd874acdd, 9407d8f24453477abb5933a3964bcfd6; no voters:
I20260927 02:44:34.653496 8444 raft_consensus.cc:2781] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:34.653613 8444 raft_consensus.cc:686] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 LEADER]: Becoming Leader. State: Replica: 5330fab27c8b4c3b91adea7dd874acdd, State: Running, Role: LEADER
I20260927 02:44:34.653750 8444 consensus_queue.cc:237] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.655293 7914 catalog_manager.cc:5495] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5330fab27c8b4c3b91adea7dd874acdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:44:34 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:34.721490 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.713552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33527 (local address 127.0.0.1:40224)
0927 02:44:34.713856 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.713870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.713967 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.714216 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.714218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.714683 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:34.714772 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.714778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.715590 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.715593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.716213 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.716221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.716626 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.716628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.716687 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.717260 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:34.717272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:34.718599 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:34.720144 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.720152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.720156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.720371 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.720599 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:34.720601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:34.720602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:34.720673 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:34.721034 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:34.721039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:34.721136 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.721223 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.721332 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260927 02:44:34.721513 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.713597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.0.98.62:33527)
0927 02:44:34.713746 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.713751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.713947 (+ 196us) server_negotiation.cc:407] Connection header received
0927 02:44:34.714039 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.714042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.714115 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.714183 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:34.714932 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.715498 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.716322 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.716533 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.716609 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.718724 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:34.718727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:34.718729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:34.718749 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:34.720054 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.720452 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.720454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.720455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.720522 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:34.720744 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:34.720746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:34.720747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:34.720947 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:34.721022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.721276 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.721384 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":60}
I20260927 02:44:34.725553 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40224:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:44:34.726374 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:34.731225 8191 tablet_service.cc:1467] Processing CreateTablet for tablet af55e3a533d348f59c367d91764c2f66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73a9c5019310439b9d7152adc4833cf7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:34.731160 8035 tablet_service.cc:1467] Processing CreateTablet for tablet af55e3a533d348f59c367d91764c2f66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73a9c5019310439b9d7152adc4833cf7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:34.731381 8331 tablet_service.cc:1467] Processing CreateTablet for tablet af55e3a533d348f59c367d91764c2f66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73a9c5019310439b9d7152adc4833cf7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:44:34.731662 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af55e3a533d348f59c367d91764c2f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.731662 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af55e3a533d348f59c367d91764c2f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.731760 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af55e3a533d348f59c367d91764c2f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:34.734493 8425 tablet_bootstrap.cc:492] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: Bootstrap starting.
I20260927 02:44:34.734512 8426 tablet_bootstrap.cc:492] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Bootstrap starting.
I20260927 02:44:34.735342 8426 tablet_bootstrap.cc:654] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.735406 8425 tablet_bootstrap.cc:654] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.736038 8427 tablet_bootstrap.cc:492] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: Bootstrap starting.
I20260927 02:44:34.736778 8425 tablet_bootstrap.cc:492] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: No bootstrap required, opened a new log
I20260927 02:44:34.736871 8425 ts_tablet_manager.cc:1397] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:34.736917 8427 tablet_bootstrap.cc:654] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:34.737491 8426 tablet_bootstrap.cc:492] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: No bootstrap required, opened a new log
I20260927 02:44:34.737509 8425 raft_consensus.cc:348] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.737597 8426 ts_tablet_manager.cc:1397] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:44:34.737637 8425 raft_consensus.cc:374] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.737676 8425 raft_consensus.cc:729] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9407d8f24453477abb5933a3964bcfd6, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.737802 8425 consensus_queue.cc:260] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.737962 8427 tablet_bootstrap.cc:492] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: No bootstrap required, opened a new log
I20260927 02:44:34.738044 8427 ts_tablet_manager.cc:1397] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:34.738087 8425 ts_tablet_manager.cc:1428] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:34.738060 8426 raft_consensus.cc:348] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.738189 8426 raft_consensus.cc:374] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.738230 8426 raft_consensus.cc:729] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2c26b1fc634ca5a12fd8e9c5e70e04, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.738363 8426 consensus_queue.cc:260] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.738451 8427 raft_consensus.cc:348] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.738562 8427 raft_consensus.cc:374] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:34.738601 8427 raft_consensus.cc:729] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5330fab27c8b4c3b91adea7dd874acdd, State: Initialized, Role: FOLLOWER
I20260927 02:44:34.738654 8426 ts_tablet_manager.cc:1428] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:34.738722 8427 consensus_queue.cc:260] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.738867 8432 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.739009 8432 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:34.739032 8427 ts_tablet_manager.cc:1428] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:34.738996 8444 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.738996 8431 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.739113 8444 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:34.739125 8431 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 02:44:34.741077 8398 tablet.cc:2369] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:34.871001 8432 raft_consensus.cc:482] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:34.871150 8432 raft_consensus.cc:504] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.872241 8432 leader_election.cc:288] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537), 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:34.876137 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.872633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48011 (local address 127.0.98.3:37329)
0927 02:44:34.872763 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.872768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.872996 (+ 228us) server_negotiation.cc:407] Connection header received
0927 02:44:34.873148 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.873150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.873213 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.873289 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:34.873558 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.874063 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.875174 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.875561 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.875639 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.875722 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.875919 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.876010 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:44:34.876163 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.872567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51447 (local address 127.0.98.1:40537)
0927 02:44:34.872700 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.872704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.872912 (+ 208us) server_negotiation.cc:407] Connection header received
0927 02:44:34.873062 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.873068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.873126 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.873199 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:34.873533 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.874031 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.875215 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.875595 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.875717 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.875784 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.875996 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.876062 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:44:34.876186 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.872460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40537 (local address 127.0.98.2:51447)
0927 02:44:34.872802 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.872824 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.872923 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.873233 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.873239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.873260 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:34.873402 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.873410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.874142 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.874148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.875062 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.875072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.875738 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.875744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.875838 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.875878 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.875979 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.876044 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":62,"thread_start_us":110,"threads_started":1}
I20260927 02:44:34.876163 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.872464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37329 (local address 127.0.98.2:48011)
0927 02:44:34.872890 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.872904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.873043 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.873317 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.873320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.873336 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:34.873432 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.873437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.874200 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.874202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.875047 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.875056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.875677 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.875684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.875779 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.875808 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.875899 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.876001 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":128,"thread_start_us":76,"threads_started":1}
I20260927 02:44:34.876711 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af55e3a533d348f59c367d91764c2f66" candidate_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5330fab27c8b4c3b91adea7dd874acdd" is_pre_election: true
I20260927 02:44:34.876892 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af55e3a533d348f59c367d91764c2f66" candidate_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9407d8f24453477abb5933a3964bcfd6" is_pre_election: true
I20260927 02:44:34.876976 8071 raft_consensus.cc:2454] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef2c26b1fc634ca5a12fd8e9c5e70e04 in term 0.
I20260927 02:44:34.877056 8351 raft_consensus.cc:2454] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef2c26b1fc634ca5a12fd8e9c5e70e04 in term 0.
I20260927 02:44:34.877403 8143 leader_election.cc:302] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9407d8f24453477abb5933a3964bcfd6, ef2c26b1fc634ca5a12fd8e9c5e70e04; no voters:
I20260927 02:44:34.877655 8432 raft_consensus.cc:2781] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:34.877712 8432 raft_consensus.cc:482] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:34.877756 8432 raft_consensus.cc:3037] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.878866 8432 raft_consensus.cc:504] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.879220 8432 leader_election.cc:288] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [CANDIDATE]: Term 1 election: Requested vote from peers 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537), 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:34.879498 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af55e3a533d348f59c367d91764c2f66" candidate_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5330fab27c8b4c3b91adea7dd874acdd"
I20260927 02:44:34.879638 8071 raft_consensus.cc:3037] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.879627 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af55e3a533d348f59c367d91764c2f66" candidate_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9407d8f24453477abb5933a3964bcfd6"
I20260927 02:44:34.879741 8351 raft_consensus.cc:3037] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:34.880780 8071 raft_consensus.cc:2454] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef2c26b1fc634ca5a12fd8e9c5e70e04 in term 1.
I20260927 02:44:34.880780 8351 raft_consensus.cc:2454] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef2c26b1fc634ca5a12fd8e9c5e70e04 in term 1.
I20260927 02:44:34.881047 8143 leader_election.cc:302] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9407d8f24453477abb5933a3964bcfd6, ef2c26b1fc634ca5a12fd8e9c5e70e04; no voters:
I20260927 02:44:34.881206 8432 raft_consensus.cc:2781] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:34.881386 8432 raft_consensus.cc:686] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 LEADER]: Becoming Leader. State: Replica: ef2c26b1fc634ca5a12fd8e9c5e70e04, State: Running, Role: LEADER
I20260927 02:44:34.881625 8432 consensus_queue.cc:237] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:34.882803 8467 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: TxnStatusTablet state changed. Reason: New leader ef2c26b1fc634ca5a12fd8e9c5e70e04. Latest consensus state: current_term: 1 leader_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.882936 8467 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:34.883196 8469 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:44:34.883322 8469 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:44:34.884158 7914 catalog_manager.cc:5495] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:34.930541 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.926365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42639 (local address 127.0.0.1:46884)
0927 02:44:34.926597 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.926610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.926725 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.927084 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.927086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.927093 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:34.927180 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.927184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.928236 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.928239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.928849 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.928857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.929419 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.929422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.929485 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.929526 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:34.930292 (+ 766us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:34.930305 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.930363 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.930432 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 02:44:34.930557 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.926519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.0.98.2:42639)
0927 02:44:34.926663 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.926668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.926684 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:34.926883 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.926887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.926953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.927060 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:34.927278 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.928111 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.928965 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.929311 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.929412 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.929664 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:34.930193 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:34.930302 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.930377 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.930428 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:44:34.933319 8351 raft_consensus.cc:1261] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Refusing update from remote peer ef2c26b1fc634ca5a12fd8e9c5e70e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:34.933467 8071 raft_consensus.cc:1261] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Refusing update from remote peer ef2c26b1fc634ca5a12fd8e9c5e70e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:34.933912 8432 consensus_queue.cc:1035] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:34.934079 8467 consensus_queue.cc:1035] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:34.935660 8444 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: TxnStatusTablet state changed. Reason: New leader ef2c26b1fc634ca5a12fd8e9c5e70e04. Latest consensus state: current_term: 1 leader_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.935770 8444 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:34.936421 8431 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: TxnStatusTablet state changed. Reason: New leader ef2c26b1fc634ca5a12fd8e9c5e70e04. Latest consensus state: current_term: 1 leader_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.936568 8431 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:34.936972 8467 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.937085 8467 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:34.938758 8431 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.938861 8431 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P 9407d8f24453477abb5933a3964bcfd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:34.942075 8432 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.942175 8432 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P ef2c26b1fc634ca5a12fd8e9c5e70e04: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:44:34.943041 8439 mvcc.cc:205] Tried to move back new op lower bound from 7333794098921926656 to 7333794098717380608. Current Snapshot: MvccSnapshot[applied={T|T < 7333794098921926656}]
I20260927 02:44:34.943192 8444 tablet_replica.cc:439] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } } }
I20260927 02:44:34.943282 8444 tablet_replica.cc:442] T af55e3a533d348f59c367d91764c2f66 P 5330fab27c8b4c3b91adea7dd874acdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:44:34.951776 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.948038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42639 (local address 127.0.98.1:49857)
0927 02:44:34.948384 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.948397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.948480 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.948698 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.948700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.948706 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:34.948841 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.948846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.949870 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.949873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.950497 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.950506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.951096 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.951099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.951174 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.951213 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:34.951540 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:34.951546 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.951599 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.951668 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":103,"threads_started":1}
I20260927 02:44:34.951779 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.948142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49857 (local address 127.0.98.2:42639)
0927 02:44:34.948259 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.948264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.948461 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:44:34.948547 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.948549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.948606 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.948685 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:34.948940 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.949724 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.950612 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.950954 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.951065 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.951380 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:34.951455 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:34.951526 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.951609 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.951657 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:44:34.957625 8211 raft_consensus.cc:1261] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Refusing update from remote peer 5330fab27c8b4c3b91adea7dd874acdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:34.957755 8351 raft_consensus.cc:1261] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Refusing update from remote peer 5330fab27c8b4c3b91adea7dd874acdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:34.958082 8433 consensus_queue.cc:1035] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:34.958233 8444 consensus_queue.cc:1035] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:34.981945 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.977787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57769 (local address 127.0.98.1:40537)
0927 02:44:34.977897 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:34.977901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:34.977914 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:34.978096 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:34.978100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:34.978147 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:34.978221 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:34.978649 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.979455 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.980694 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.981063 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.981135 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.981507 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:34.981593 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:34.981646 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:34.981793 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 02:44:34.981842 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:44:34.982002 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:34.977377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40537 (local address 127.0.98.2:57769)
0927 02:44:34.977678 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:34.977693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:34.977784 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:34.978242 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:34.978245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:34.978254 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:44:34.978379 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.978385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.979610 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.979614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.980553 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:34.980565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:34.981249 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:34.981252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:34.981331 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:34.981373 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:44:34.981707 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:44:34.981712 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:44:34.981771 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:44:34.981843 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
I20260927 02:44:35.004679 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:35.004846 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:35.006893 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: Sending DeleteTablet for 3 replicas of tablet e83d0332c8f6486a937217401b1d5d53
I20260927 02:44:35.007434 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet e83d0332c8f6486a937217401b1d5d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39162
I20260927 02:44:35.007583 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet e83d0332c8f6486a937217401b1d5d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34048
I20260927 02:44:35.008090 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet e83d0332c8f6486a937217401b1d5d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46872
I20260927 02:44:35.008179 8483 tablet_replica.cc:330] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: stopping tablet replica
I20260927 02:44:35.008241 8451 tablet_replica.cc:330] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: stopping tablet replica
I20260927 02:44:35.008337 8451 raft_consensus.cc:2229] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:35.008332 8483 raft_consensus.cc:2229] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:35.008430 8451 raft_consensus.cc:2258] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.008447 8483 raft_consensus.cc:2258] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.009053 8451 ts_tablet_manager.cc:1905] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:35.011402 8482 tablet_replica.cc:330] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: stopping tablet replica
I20260927 02:44:35.011567 8482 raft_consensus.cc:2229] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:35.011775 8482 raft_consensus.cc:2258] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.011813 8483 ts_tablet_manager.cc:1905] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:35.011962 8451 ts_tablet_manager.cc:1918] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:35.012053 8451 log.cc:1196] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/e83d0332c8f6486a937217401b1d5d53
I20260927 02:44:35.012327 8451 ts_tablet_manager.cc:1939] T e83d0332c8f6486a937217401b1d5d53 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting consensus metadata
I20260927 02:44:35.012522 8482 ts_tablet_manager.cc:1905] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:35.013192 7899 catalog_manager.cc:4841] TS ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639): tablet e83d0332c8f6486a937217401b1d5d53 (table test-table [id=849203b6d72e4016ad153145326f7404]) successfully deleted
I20260927 02:44:35.014559 8483 ts_tablet_manager.cc:1918] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:35.014643 8483 log.cc:1196] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/e83d0332c8f6486a937217401b1d5d53
I20260927 02:44:35.014891 8483 ts_tablet_manager.cc:1939] T e83d0332c8f6486a937217401b1d5d53 P 9407d8f24453477abb5933a3964bcfd6: Deleting consensus metadata
I20260927 02:44:35.015522 7899 catalog_manager.cc:4841] TS 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329): tablet e83d0332c8f6486a937217401b1d5d53 (table test-table [id=849203b6d72e4016ad153145326f7404]) successfully deleted
I20260927 02:44:35.015808 8482 ts_tablet_manager.cc:1918] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:35.015914 8482 log.cc:1196] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/e83d0332c8f6486a937217401b1d5d53
I20260927 02:44:35.016160 8482 ts_tablet_manager.cc:1939] T e83d0332c8f6486a937217401b1d5d53 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting consensus metadata
I20260927 02:44:35.016736 7898 catalog_manager.cc:4841] TS 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537): tablet e83d0332c8f6486a937217401b1d5d53 (table test-table [id=849203b6d72e4016ad153145326f7404]) successfully deleted
Sep 27 02:44:35 dist-test-slave-gkrl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:44:35 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:35 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:44:35.037163 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:35.030299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.0.98.62:33527)
0927 02:44:35.030454 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:35.030460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:35.030638 (+ 178us) server_negotiation.cc:407] Connection header received
0927 02:44:35.030889 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:35.030892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:35.030940 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:35.031036 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:35.031714 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.032331 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.033209 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.033436 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.033553 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:35.037051 (+ 3498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40230: BlockingRecv error: recv got EOF from 127.0.0.1:40230 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 02:44:35 dist-test-slave-gkrl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:44:35.049234 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:35.038996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.0.98.62:33527)
0927 02:44:35.039092 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:35.039096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:35.039490 (+ 394us) server_negotiation.cc:407] Connection header received
0927 02:44:35.039575 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:35.039578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:35.039623 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:35.039703 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:35.040491 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.041322 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.042432 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.042779 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.042877 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:35.045677 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:35.045680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:35.045683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:35.045708 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:35.047415 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:35.047937 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:35.047941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:35.047943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:35.048026 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:35.048336 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:35.048341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:35.048344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:35.048551 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:35.048666 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:35.048979 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 02:44:35.049082 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:44:35.052142 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:35.052567 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:35.057101 8035 tablet_service.cc:1467] Processing CreateTablet for tablet a78f6121a6ff43bb9f00e46885353788 (DEFAULT_TABLE table=test-table [id=63e1b30f06724bb6bf01a78285a66e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:35.057374 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78f6121a6ff43bb9f00e46885353788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:35.057345 8191 tablet_service.cc:1467] Processing CreateTablet for tablet a78f6121a6ff43bb9f00e46885353788 (DEFAULT_TABLE table=test-table [id=63e1b30f06724bb6bf01a78285a66e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:35.057577 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78f6121a6ff43bb9f00e46885353788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:35.059510 8426 tablet_bootstrap.cc:492] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Bootstrap starting.
I20260927 02:44:35.059729 8427 tablet_bootstrap.cc:492] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: Bootstrap starting.
I20260927 02:44:35.059741 8331 tablet_service.cc:1467] Processing CreateTablet for tablet a78f6121a6ff43bb9f00e46885353788 (DEFAULT_TABLE table=test-table [id=63e1b30f06724bb6bf01a78285a66e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:35.060050 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78f6121a6ff43bb9f00e46885353788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:35.060395 8426 tablet_bootstrap.cc:654] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:35.060637 8427 tablet_bootstrap.cc:654] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:35.061744 8427 tablet_bootstrap.cc:492] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: No bootstrap required, opened a new log
I20260927 02:44:35.061832 8427 ts_tablet_manager.cc:1397] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:35.062213 8425 tablet_bootstrap.cc:492] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: Bootstrap starting.
I20260927 02:44:35.062297 8427 raft_consensus.cc:348] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.062410 8427 raft_consensus.cc:374] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:35.062455 8427 raft_consensus.cc:729] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5330fab27c8b4c3b91adea7dd874acdd, State: Initialized, Role: FOLLOWER
I20260927 02:44:35.062587 8427 consensus_queue.cc:260] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.062865 8427 ts_tablet_manager.cc:1428] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:35.062880 8426 tablet_bootstrap.cc:492] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: No bootstrap required, opened a new log
I20260927 02:44:35.062958 8426 ts_tablet_manager.cc:1397] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:44:35.063160 8425 tablet_bootstrap.cc:654] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:35.063313 8426 raft_consensus.cc:348] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.063434 8426 raft_consensus.cc:374] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:35.063488 8426 raft_consensus.cc:729] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2c26b1fc634ca5a12fd8e9c5e70e04, State: Initialized, Role: FOLLOWER
I20260927 02:44:35.063625 8426 consensus_queue.cc:260] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.063918 8426 ts_tablet_manager.cc:1428] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:35.065021 8425 tablet_bootstrap.cc:492] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: No bootstrap required, opened a new log
I20260927 02:44:35.065119 8425 ts_tablet_manager.cc:1397] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:35.066039 8425 raft_consensus.cc:348] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.066170 8425 raft_consensus.cc:374] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:35.066205 8425 raft_consensus.cc:729] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9407d8f24453477abb5933a3964bcfd6, State: Initialized, Role: FOLLOWER
I20260927 02:44:35.066339 8425 consensus_queue.cc:260] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.066597 8425 ts_tablet_manager.cc:1428] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:35.204460 8431 raft_consensus.cc:482] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:35.204627 8431 raft_consensus.cc:504] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.205739 8431 leader_election.cc:288] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537), ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639)
I20260927 02:44:35.209667 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:35.206062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47477 (local address 127.0.98.1:40537)
0927 02:44:35.206212 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:35.206217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:35.206366 (+ 149us) server_negotiation.cc:407] Connection header received
0927 02:44:35.206550 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:35.206553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:35.206598 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:35.206681 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:35.207002 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.207522 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.208728 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.209124 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.209243 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:35.209307 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:35.209477 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 02:44:35.209547 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:35.209693 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:35.205996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40537 (local address 127.0.98.3:47477)
0927 02:44:35.206273 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:35.206286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:35.206404 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:35.206693 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:35.206695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:35.206712 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:35.206837 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:35.206843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.207638 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.207642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:35.208599 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:35.208608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.209245 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.209248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:35.209349 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:35.209368 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:44:35.209460 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:44:35.209528 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":54,"thread_start_us":87,"threads_started":1}
I20260927 02:44:35.209703 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:35.206120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33649 (local address 127.0.98.2:42639)
0927 02:44:35.206241 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:35.206245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:35.206628 (+ 383us) server_negotiation.cc:407] Connection header received
0927 02:44:35.206777 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:35.206780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:35.206831 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:35.206923 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:35.207178 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.207678 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.208766 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.209158 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.209286 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:35.209346 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:35.209528 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 02:44:35.209589 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:44:35.209730 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:35.206000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42639 (local address 127.0.98.3:33649)
0927 02:44:35.206542 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:35.206553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:35.206670 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:35.206942 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:35.206944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:35.206960 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:35.207068 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:35.207073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.207802 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.207805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:35.208647 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:35.208655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.209286 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.209290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:35.209399 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:35.209420 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:44:35.209512 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:44:35.209578 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":254,"thread_start_us":96,"threads_started":1}
I20260927 02:44:35.210340 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78f6121a6ff43bb9f00e46885353788" candidate_uuid: "9407d8f24453477abb5933a3964bcfd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5330fab27c8b4c3b91adea7dd874acdd" is_pre_election: true
I20260927 02:44:35.210484 8071 raft_consensus.cc:2454] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9407d8f24453477abb5933a3964bcfd6 in term 0.
I20260927 02:44:35.210425 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78f6121a6ff43bb9f00e46885353788" candidate_uuid: "9407d8f24453477abb5933a3964bcfd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" is_pre_election: true
I20260927 02:44:35.210632 8211 raft_consensus.cc:2454] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9407d8f24453477abb5933a3964bcfd6 in term 0.
I20260927 02:44:35.210872 8282 leader_election.cc:302] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5330fab27c8b4c3b91adea7dd874acdd, 9407d8f24453477abb5933a3964bcfd6; no voters:
I20260927 02:44:35.211128 8431 raft_consensus.cc:2781] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:35.211194 8431 raft_consensus.cc:482] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:35.211232 8431 raft_consensus.cc:3037] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:35.212204 8431 raft_consensus.cc:504] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.212531 8431 leader_election.cc:288] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [CANDIDATE]: Term 1 election: Requested vote from peers 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537), ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639)
I20260927 02:44:35.212958 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78f6121a6ff43bb9f00e46885353788" candidate_uuid: "9407d8f24453477abb5933a3964bcfd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5330fab27c8b4c3b91adea7dd874acdd"
I20260927 02:44:35.213009 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78f6121a6ff43bb9f00e46885353788" candidate_uuid: "9407d8f24453477abb5933a3964bcfd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04"
I20260927 02:44:35.213081 8071 raft_consensus.cc:3037] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:35.213131 8211 raft_consensus.cc:3037] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:35.214139 8071 raft_consensus.cc:2454] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9407d8f24453477abb5933a3964bcfd6 in term 1.
I20260927 02:44:35.214169 8211 raft_consensus.cc:2454] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9407d8f24453477abb5933a3964bcfd6 in term 1.
I20260927 02:44:35.214433 8282 leader_election.cc:302] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5330fab27c8b4c3b91adea7dd874acdd, 9407d8f24453477abb5933a3964bcfd6; no voters:
I20260927 02:44:35.214658 8431 raft_consensus.cc:2781] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:35.214840 8431 raft_consensus.cc:686] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 1 LEADER]: Becoming Leader. State: Replica: 9407d8f24453477abb5933a3964bcfd6, State: Running, Role: LEADER
I20260927 02:44:35.215101 8431 consensus_queue.cc:237] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.217552 7912 catalog_manager.cc:5495] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9407d8f24453477abb5933a3964bcfd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } health_report { overall_health: HEALTHY } } }
I20260927 02:44:35.257982 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:35.254097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.0.98.3:37329)
0927 02:44:35.254223 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:35.254227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:35.254241 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:35.254329 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:35.254331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:35.254379 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:35.254465 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:35.254701 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.255201 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.256435 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.256655 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.256771 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:35.257070 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:35.257571 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:35.257667 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:35.257741 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:44:35.257802 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:44:35.260667 8211 raft_consensus.cc:1261] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Refusing update from remote peer 9407d8f24453477abb5933a3964bcfd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:35.260725 8071 raft_consensus.cc:1261] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Refusing update from remote peer 9407d8f24453477abb5933a3964bcfd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:35.261209 8431 consensus_queue.cc:1035] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:35.261408 8501 consensus_queue.cc:1035] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:35.271799 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:35.271989 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:35.274778 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: Sending DeleteTablet for 3 replicas of tablet a78f6121a6ff43bb9f00e46885353788
I20260927 02:44:35.275488 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet a78f6121a6ff43bb9f00e46885353788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39162
I20260927 02:44:35.276057 8482 tablet_replica.cc:330] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: stopping tablet replica
I20260927 02:44:35.276204 8482 raft_consensus.cc:2229] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:35.276357 8482 raft_consensus.cc:2258] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.275653 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet a78f6121a6ff43bb9f00e46885353788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46872
I20260927 02:44:35.276688 8451 tablet_replica.cc:330] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: stopping tablet replica
I20260927 02:44:35.276829 8451 raft_consensus.cc:2229] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:35.276973 8451 raft_consensus.cc:2258] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.275705 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet a78f6121a6ff43bb9f00e46885353788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34048
I20260927 02:44:35.277449 8483 tablet_replica.cc:330] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: stopping tablet replica
I20260927 02:44:35.277586 8483 raft_consensus.cc:2229] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:35.277927 8483 raft_consensus.cc:2258] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.278985 8483 ts_tablet_manager.cc:1905] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:35.279206 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:35.279644 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:35.280808 8482 ts_tablet_manager.cc:1905] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:35.281934 8483 ts_tablet_manager.cc:1918] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:35.282023 8483 log.cc:1196] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/a78f6121a6ff43bb9f00e46885353788
I20260927 02:44:35.282347 8483 ts_tablet_manager.cc:1939] T a78f6121a6ff43bb9f00e46885353788 P 9407d8f24453477abb5933a3964bcfd6: Deleting consensus metadata
I20260927 02:44:35.283075 7899 catalog_manager.cc:4841] TS 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329): tablet a78f6121a6ff43bb9f00e46885353788 (table test-table [id=63e1b30f06724bb6bf01a78285a66e36]) successfully deleted
I20260927 02:44:35.283329 8482 ts_tablet_manager.cc:1918] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:35.283397 8482 log.cc:1196] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/a78f6121a6ff43bb9f00e46885353788
I20260927 02:44:35.283588 8482 ts_tablet_manager.cc:1939] T a78f6121a6ff43bb9f00e46885353788 P 5330fab27c8b4c3b91adea7dd874acdd: Deleting consensus metadata
I20260927 02:44:35.284358 7898 catalog_manager.cc:4841] TS 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537): tablet a78f6121a6ff43bb9f00e46885353788 (table test-table [id=63e1b30f06724bb6bf01a78285a66e36]) successfully deleted
I20260927 02:44:35.286062 8451 ts_tablet_manager.cc:1905] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:35.288220 8035 tablet_service.cc:1467] Processing CreateTablet for tablet 89938b7605c24633bc6ea2d9b952c5cd (DEFAULT_TABLE table=test-table [id=fdfd63f812ec4a5494d6bc9f2493e478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:35.288338 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 89938b7605c24633bc6ea2d9b952c5cd (DEFAULT_TABLE table=test-table [id=fdfd63f812ec4a5494d6bc9f2493e478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:35.288522 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89938b7605c24633bc6ea2d9b952c5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:35.288596 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89938b7605c24633bc6ea2d9b952c5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:35.288787 8451 ts_tablet_manager.cc:1918] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:35.288854 8451 log.cc:1196] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/a78f6121a6ff43bb9f00e46885353788
I20260927 02:44:35.289121 8451 ts_tablet_manager.cc:1939] T a78f6121a6ff43bb9f00e46885353788 P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting consensus metadata
I20260927 02:44:35.289835 7899 catalog_manager.cc:4841] TS ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639): tablet a78f6121a6ff43bb9f00e46885353788 (table test-table [id=63e1b30f06724bb6bf01a78285a66e36]) successfully deleted
I20260927 02:44:35.290632 8427 tablet_bootstrap.cc:492] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd: Bootstrap starting.
I20260927 02:44:35.290707 8426 tablet_bootstrap.cc:492] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: Bootstrap starting.
I20260927 02:44:35.291389 8427 tablet_bootstrap.cc:654] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:35.291590 8426 tablet_bootstrap.cc:654] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:35.291982 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 89938b7605c24633bc6ea2d9b952c5cd (DEFAULT_TABLE table=test-table [id=fdfd63f812ec4a5494d6bc9f2493e478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:35.292253 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89938b7605c24633bc6ea2d9b952c5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:35.292454 8427 tablet_bootstrap.cc:492] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd: No bootstrap required, opened a new log
I20260927 02:44:35.292520 8427 ts_tablet_manager.cc:1397] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:35.292686 8426 tablet_bootstrap.cc:492] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: No bootstrap required, opened a new log
I20260927 02:44:35.292763 8426 ts_tablet_manager.cc:1397] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:35.292865 8427 raft_consensus.cc:348] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.292964 8427 raft_consensus.cc:374] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:35.292994 8427 raft_consensus.cc:729] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5330fab27c8b4c3b91adea7dd874acdd, State: Initialized, Role: FOLLOWER
I20260927 02:44:35.293100 8427 consensus_queue.cc:260] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.293130 8426 raft_consensus.cc:348] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.293260 8426 raft_consensus.cc:374] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:35.293299 8426 raft_consensus.cc:729] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2c26b1fc634ca5a12fd8e9c5e70e04, State: Initialized, Role: FOLLOWER
I20260927 02:44:35.293339 8427 ts_tablet_manager.cc:1428] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:35.293442 8426 consensus_queue.cc:260] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.293728 8426 ts_tablet_manager.cc:1428] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:35.294528 8425 tablet_bootstrap.cc:492] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: Bootstrap starting.
I20260927 02:44:35.295490 8425 tablet_bootstrap.cc:654] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:35.296911 8425 tablet_bootstrap.cc:492] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: No bootstrap required, opened a new log
I20260927 02:44:35.297000 8425 ts_tablet_manager.cc:1397] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:35.297513 8425 raft_consensus.cc:348] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.297636 8425 raft_consensus.cc:374] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:35.297685 8425 raft_consensus.cc:729] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9407d8f24453477abb5933a3964bcfd6, State: Initialized, Role: FOLLOWER
I20260927 02:44:35.297818 8425 consensus_queue.cc:260] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.298170 8425 ts_tablet_manager.cc:1428] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:35.461339 8436 raft_consensus.cc:482] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:35.461503 8436 raft_consensus.cc:504] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.462000 8436 leader_election.cc:288] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639), 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:35.462535 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938b7605c24633bc6ea2d9b952c5cd" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" is_pre_election: true
I20260927 02:44:35.462560 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938b7605c24633bc6ea2d9b952c5cd" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9407d8f24453477abb5933a3964bcfd6" is_pre_election: true
I20260927 02:44:35.462718 8351 raft_consensus.cc:2454] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 0.
I20260927 02:44:35.462709 8211 raft_consensus.cc:2454] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 0.
I20260927 02:44:35.463081 8003 leader_election.cc:302] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5330fab27c8b4c3b91adea7dd874acdd, 9407d8f24453477abb5933a3964bcfd6; no voters:
I20260927 02:44:35.463297 8436 raft_consensus.cc:2781] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:35.463369 8436 raft_consensus.cc:482] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:35.463416 8436 raft_consensus.cc:3037] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:35.464532 8436 raft_consensus.cc:504] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.464882 8436 leader_election.cc:288] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 election: Requested vote from peers ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639), 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329)
I20260927 02:44:35.465277 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938b7605c24633bc6ea2d9b952c5cd" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9407d8f24453477abb5933a3964bcfd6"
I20260927 02:44:35.465291 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938b7605c24633bc6ea2d9b952c5cd" candidate_uuid: "5330fab27c8b4c3b91adea7dd874acdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04"
I20260927 02:44:35.465418 8211 raft_consensus.cc:3037] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:35.465418 8351 raft_consensus.cc:3037] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:35.466553 8351 raft_consensus.cc:2454] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 1.
I20260927 02:44:35.466553 8211 raft_consensus.cc:2454] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5330fab27c8b4c3b91adea7dd874acdd in term 1.
I20260927 02:44:35.466893 8003 leader_election.cc:302] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5330fab27c8b4c3b91adea7dd874acdd, ef2c26b1fc634ca5a12fd8e9c5e70e04; no voters:
I20260927 02:44:35.467096 8436 raft_consensus.cc:2781] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:35.467190 8436 raft_consensus.cc:686] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 1 LEADER]: Becoming Leader. State: Replica: 5330fab27c8b4c3b91adea7dd874acdd, State: Running, Role: LEADER
I20260927 02:44:35.467347 8436 consensus_queue.cc:237] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } }
I20260927 02:44:35.469055 7912 catalog_manager.cc:5495] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5330fab27c8b4c3b91adea7dd874acdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5330fab27c8b4c3b91adea7dd874acdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:35.493541 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:35.490779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.0.98.1:40537)
0927 02:44:35.490876 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:35.490879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:35.490916 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:44:35.491054 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:35.491056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:35.491095 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:35.491155 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:35.491348 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.491821 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.492639 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:35.492847 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:35.492927 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:35.493136 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:35.493198 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:35.493251 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:35.493344 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 02:44:35.493381 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:44:35.499626 8351 raft_consensus.cc:1261] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Refusing update from remote peer 5330fab27c8b4c3b91adea7dd874acdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:35.499697 8211 raft_consensus.cc:1261] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Refusing update from remote peer 5330fab27c8b4c3b91adea7dd874acdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:35.500056 8433 consensus_queue.cc:1035] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9407d8f24453477abb5933a3964bcfd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:35.500222 8436 consensus_queue.cc:1035] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2c26b1fc634ca5a12fd8e9c5e70e04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:35.529902 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:35.530035 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:35.531917 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1ecc54309a84d36900e9b008fc70581: Sending DeleteTablet for 3 replicas of tablet 89938b7605c24633bc6ea2d9b952c5cd
I20260927 02:44:35.532449 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet 89938b7605c24633bc6ea2d9b952c5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39162
I20260927 02:44:35.532491 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 89938b7605c24633bc6ea2d9b952c5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46872
I20260927 02:44:35.532634 8451 tablet_replica.cc:330] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: stopping tablet replica
I20260927 02:44:35.532655 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 89938b7605c24633bc6ea2d9b952c5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34048
I20260927 02:44:35.532671 8482 tablet_replica.cc:330] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd: stopping tablet replica
I20260927 02:44:35.532754 8451 raft_consensus.cc:2229] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:35.532806 8483 tablet_replica.cc:330] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: stopping tablet replica
I20260927 02:44:35.532847 8451 raft_consensus.cc:2258] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.532835 8482 raft_consensus.cc:2229] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:35.532920 8483 raft_consensus.cc:2229] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:35.533012 8483 raft_consensus.cc:2258] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.533006 8482 raft_consensus.cc:2258] T 89938b7605c24633bc6ea2d9b952c5cd P 5330fab27c8b4c3b91adea7dd874acdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:35.533633 8451 ts_tablet_manager.cc:1905] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:35.533835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 7986
I20260927 02:44:35.533838 8483 ts_tablet_manager.cc:1905] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:35.536239 8451 ts_tablet_manager.cc:1918] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:35.536239 8483 ts_tablet_manager.cc:1918] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:44:35.536329 8451 log.cc:1196] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/89938b7605c24633bc6ea2d9b952c5cd
I20260927 02:44:35.536329 8483 log.cc:1196] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/89938b7605c24633bc6ea2d9b952c5cd
I20260927 02:44:35.536607 8483 ts_tablet_manager.cc:1939] T 89938b7605c24633bc6ea2d9b952c5cd P 9407d8f24453477abb5933a3964bcfd6: Deleting consensus metadata
I20260927 02:44:35.536607 8451 ts_tablet_manager.cc:1939] T 89938b7605c24633bc6ea2d9b952c5cd P ef2c26b1fc634ca5a12fd8e9c5e70e04: Deleting consensus metadata
I20260927 02:44:35.537267 7899 catalog_manager.cc:4841] TS ef2c26b1fc634ca5a12fd8e9c5e70e04 (127.0.98.2:42639): tablet 89938b7605c24633bc6ea2d9b952c5cd (table test-table [id=fdfd63f812ec4a5494d6bc9f2493e478]) successfully deleted
I20260927 02:44:35.537452 7899 catalog_manager.cc:4841] TS 9407d8f24453477abb5933a3964bcfd6 (127.0.98.3:37329): tablet 89938b7605c24633bc6ea2d9b952c5cd (table test-table [id=fdfd63f812ec4a5494d6bc9f2493e478]) successfully deleted
I20260927 02:44:35.543010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 8126
W20260927 02:44:35.543045 7898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40537 (error 108)
I20260927 02:44:35.549100 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:35.548711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40537 (local address 127.0.0.1:39194)
0927 02:44:35.548943 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:35.549018 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
W20260927 02:44:35.549232 7898 catalog_manager.cc:4584] TS 5330fab27c8b4c3b91adea7dd874acdd (127.0.98.1:40537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89938b7605c24633bc6ea2d9b952c5cd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40537: connect: Connection refused (error 111)
I20260927 02:44:35.549927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 8266
I20260927 02:44:35.556537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 7882
2026-09-27T02:44:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3313 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:35 dist-test-slave-gkrl krb5kdc[8521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:35 dist-test-slave-gkrl krb5kdc[8521](info): set up 2 sockets
Sep 27 02:44:35 dist-test-slave-gkrl krb5kdc[8521](info): commencing operation
krb5kdc: starting...
W20260927 02:44:36.596668 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:36 dist-test-slave-gkrl krb5kdc[8521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:36.714335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40277
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:36.819850 8534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:36.820096 8534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:36.820134 8534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:36.824069 8534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:36.824120 8534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:36.824139 8534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:36.824155 8534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:36.824170 8534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:36.829361 8534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:36.830299 8534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:36.831283 8534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:36.839725 8542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:36.839715 8539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:36.839846 8534 server_base.cc:1034] running on GCE node
W20260927 02:44:36.839704 8540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:36.840734 8534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:36.841907 8534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:36.843098 8534 hybrid_clock.cc:648] HybridClock initialized: now 1790477076843085 us; error 25 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T02:44:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1294 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:36 dist-test-slave-gkrl krb5kdc[8547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:36 dist-test-slave-gkrl krb5kdc[8547](info): set up 2 sockets
Sep 27 02:44:36 dist-test-slave-gkrl krb5kdc[8547](info): commencing operation
krb5kdc: starting...
W20260927 02:44:37.904632 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:37 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:38.038352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:38.146015 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:38.146235 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:38.146273 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:38.150336 8560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:38.150398 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:38.150417 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:38.150434 8560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:38.150449 8560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:38.156032 8560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:38.156944 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:38.157850 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:38.165969 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.165972 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.166040 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.166184 8560 server_base.cc:1034] running on GCE node
I20260927 02:44:38.166769 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:38.167840 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:38.169040 8560 hybrid_clock.cc:648] HybridClock initialized: now 1790477078169027 us; error 25 us; skew 500 ppm
Sep 27 02:44:38 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:38.171689 8560 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:38.172564 8560 webserver.cc:460] Webserver started at http://127.0.98.62:39631/ using document root <none> and password file <none>
I20260927 02:44:38.173100 8560 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:38.173187 8560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:38.173473 8560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:38.175397 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "cfccdd481d6b4d579a524bf2e9474a8d"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "1ed2ff988a3403b413c96a1fa00ce9e2"
server_key_iv: "320b42cb9354d46add3342803d4f5a4e"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.175993 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "cfccdd481d6b4d579a524bf2e9474a8d"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "1ed2ff988a3403b413c96a1fa00ce9e2"
server_key_iv: "320b42cb9354d46add3342803d4f5a4e"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.179407 8560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 02:44:38.181700 8575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.182811 8560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:44:38.182942 8560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "cfccdd481d6b4d579a524bf2e9474a8d"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "1ed2ff988a3403b413c96a1fa00ce9e2"
server_key_iv: "320b42cb9354d46add3342803d4f5a4e"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.183044 8560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.206987 8560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.209966 8560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.210170 8560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.216954 8560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35485
I20260927 02:44:38.216979 8627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35485 every 8 connection(s)
I20260927 02:44:38.217839 8560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:38.220495 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:38.224200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 8560
I20260927 02:44:38.224335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:38.224613 392 external_mini_cluster.cc:1428] Setting key 34f8d5b2a01e299e39e340358a26c3c8
I20260927 02:44:38.226001 8628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Bootstrap starting.
I20260927 02:44:38.228302 8628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:38.229084 8628 log.cc:824] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:38.230829 8628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: No bootstrap required, opened a new log
Sep 27 02:44:38 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:38.235117 8628 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260927 02:44:38.235327 8628 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:38.235388 8628 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfccdd481d6b4d579a524bf2e9474a8d, State: Initialized, Role: FOLLOWER
I20260927 02:44:38.235853 8628 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260927 02:44:38.236063 8628 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:38.236140 8628 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:38.236243 8628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:38.237321 8628 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260927 02:44:38.237709 8628 leader_election.cc:302] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfccdd481d6b4d579a524bf2e9474a8d; no voters:
I20260927 02:44:38.238036 8628 leader_election.cc:288] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:38.238112 8633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:38.238327 8633 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 1 LEADER]: Becoming Leader. State: Replica: cfccdd481d6b4d579a524bf2e9474a8d, State: Running, Role: LEADER
I20260927 02:44:38.238632 8633 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260927 02:44:38.239235 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.226034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.0.98.62:35485)
0927 02:44:38.226455 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.226470 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.226503 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:44:38.227108 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.227123 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.227436 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.227709 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.228659 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.229687 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.230573 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.231086 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.231253 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.234080 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.234086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.234098 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.234123 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.235862 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.236420 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.236429 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.236435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.236569 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.236844 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.236847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.236849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.237374 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.237562 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.237825 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.238295 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260927 02:44:38.239464 8628 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:38.239825 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfccdd481d6b4d579a524bf2e9474a8d. Latest consensus state: current_term: 1 leader_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } }
I20260927 02:44:38.239990 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:38.240235 8634 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } }
I20260927 02:44:38.240309 8634 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:38.240427 8638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:38.243129 8638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:38.248283 8638 catalog_manager.cc:1347] Generated new cluster ID: 5d630c0864274494937df5a30b7ae3ae
I20260927 02:44:38.248387 8638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:38.256852 8638 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:38.257706 8638 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:38.269299 8638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Generated new TSK 0
I20260927 02:44:38.270077 8638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:38.303459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:38.412905 8654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:38.413156 8654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:38.413201 8654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:38.417227 8654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:38.417286 8654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:38.417346 8654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:38.422423 8654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:38.423331 8654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:38.424335 8654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:38.433553 8660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.433566 8654 server_base.cc:1034] running on GCE node
W20260927 02:44:38.433560 8659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.433652 8662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.434151 8654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:38.434805 8654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:38.435967 8654 hybrid_clock.cc:648] HybridClock initialized: now 1790477078435950 us; error 31 us; skew 500 ppm
Sep 27 02:44:38 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:38.438666 8654 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:38.439568 8654 webserver.cc:460] Webserver started at http://127.0.98.1:34095/ using document root <none> and password file <none>
I20260927 02:44:38.440193 8654 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:38.440281 8654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:38.440483 8654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:38.442409 8654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "96d6d84d0c3f4934b9a2d31b7d9ae7cd"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "f7ac19c65731b2437d6e26f2e720d0fa"
server_key_iv: "f0304cac90e34be90e7e16631f5e4c84"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.442945 8654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "96d6d84d0c3f4934b9a2d31b7d9ae7cd"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "f7ac19c65731b2437d6e26f2e720d0fa"
server_key_iv: "f0304cac90e34be90e7e16631f5e4c84"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.446504 8654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:44:38.448925 8669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.450167 8654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:38.450315 8654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "96d6d84d0c3f4934b9a2d31b7d9ae7cd"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "f7ac19c65731b2437d6e26f2e720d0fa"
server_key_iv: "f0304cac90e34be90e7e16631f5e4c84"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.450420 8654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.466030 8654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.468875 8654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.469091 8654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.469655 8654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:38.470436 8654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:38.470505 8654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.470566 8654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:38.470614 8654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.479365 8654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34989
I20260927 02:44:38.479389 8782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34989 every 8 connection(s)
I20260927 02:44:38.480280 8654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:44:38 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:38.489559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 8654
I20260927 02:44:38.489708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:38.489997 392 external_mini_cluster.cc:1428] Setting key dd8633ec7d1b986957440cd8cd0afad0
I20260927 02:44:38.493808 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.481964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34769 (local address 127.0.98.62:35485)
0927 02:44:38.482096 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.482099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.482933 (+ 834us) server_negotiation.cc:407] Connection header received
0927 02:44:38.484350 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.484353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.484428 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.484529 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.485605 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.486211 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.487349 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.487595 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.487708 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.490143 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.490147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.490150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.490182 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.491709 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.492218 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.492224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.492226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.492306 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.492643 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.492647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.492648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.492897 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.493012 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.493467 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.493601 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:44:38.494443 8785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.482323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:34769)
0927 02:44:38.482744 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:38.482794 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:38.484154 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:38.484689 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:38.484695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:38.485032 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:38.485462 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.485474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.486336 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.486339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.487227 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.487235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.487740 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.487744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.487939 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.488517 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:38.488532 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:38.489988 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:38.491820 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.491826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.491839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.492117 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.492399 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.492410 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.492413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.492544 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.493020 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:38.493024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:38.493232 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 02:44:38.493389 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:44:38.493612 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260927 02:44:38.495787 8783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260927 02:44:38.496048 8783 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.496506 8783 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260927 02:44:38.498088 8592 ts_manager.cc:194] Registered new tserver with Master: 96d6d84d0c3f4934b9a2d31b7d9ae7cd (127.0.98.1:34989)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:44:38.499699 8592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:38.517009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:38.622334 8788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:38.622568 8788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:38.622608 8788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:38.626554 8788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:38.626605 8788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:38.626662 8788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:38.631695 8788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:38.632655 8788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:38.633564 8788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:38.642894 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.643033 8788 server_base.cc:1034] running on GCE node
W20260927 02:44:38.642900 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.642956 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.643568 8788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:38.644225 8788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:38.645387 8788 hybrid_clock.cc:648] HybridClock initialized: now 1790477078645350 us; error 51 us; skew 500 ppm
Sep 27 02:44:38 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:38.648212 8788 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:38.649140 8788 webserver.cc:460] Webserver started at http://127.0.98.2:44915/ using document root <none> and password file <none>
I20260927 02:44:38.649712 8788 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:38.649776 8788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:38.649987 8788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:38.651995 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4968dd4ea264483ae0875a1d9239cdb"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "6340a17c853f6ca4ba370d8497eb551b"
server_key_iv: "30404ceab5a55f9afe7f669fd19cd1b5"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.652541 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4968dd4ea264483ae0875a1d9239cdb"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "6340a17c853f6ca4ba370d8497eb551b"
server_key_iv: "30404ceab5a55f9afe7f669fd19cd1b5"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.656172 8788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:38.658502 8803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.659818 8788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:38.659998 8788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "c4968dd4ea264483ae0875a1d9239cdb"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "6340a17c853f6ca4ba370d8497eb551b"
server_key_iv: "30404ceab5a55f9afe7f669fd19cd1b5"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.660106 8788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.672580 8788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.675189 8788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.675384 8788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.675992 8788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:38.676746 8788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:38.676800 8788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.676836 8788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:38.676852 8788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.685776 8788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38327
I20260927 02:44:38.685794 8916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38327 every 8 connection(s)
I20260927 02:44:38.686666 8788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:38.692368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 8788
I20260927 02:44:38.692494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:38.692786 392 external_mini_cluster.cc:1428] Setting key 496a8b56af15468e901d27aebdc17f31
Sep 27 02:44:38 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:38.700744 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.688525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32875 (local address 127.0.98.62:35485)
0927 02:44:38.688651 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.688654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.689486 (+ 832us) server_negotiation.cc:407] Connection header received
0927 02:44:38.690766 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.690769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.690814 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.690889 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.692076 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.692709 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.694114 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.694423 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.694536 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.697279 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.697282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.697285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.697311 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.698749 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.699251 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.699256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.699258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.699343 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.699673 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.699678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.699680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.699859 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.700002 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.700468 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.700580 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:44:38.701493 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.688915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:32875)
0927 02:44:38.689324 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:38.689363 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:38.690571 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:38.691048 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:38.691054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:38.691385 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:38.691868 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.691940 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.692834 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.692838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.693997 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.694006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.694579 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.694582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.694718 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.695320 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:38.695335 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:38.697129 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:38.698868 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.698874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.698888 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.699146 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.699440 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.699443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.699445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.699579 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.700009 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:38.700018 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:38.700245 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 02:44:38.700432 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 02:44:38.700720 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:38.702622 8917 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260927 02:44:38.702811 8917 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.703264 8917 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260927 02:44:38.704331 8592 ts_manager.cc:194] Registered new tserver with Master: c4968dd4ea264483ae0875a1d9239cdb (127.0.98.2:38327)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:44:38.704981 8592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:38.719518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:38.827204 8922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:38.827436 8922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:38.827477 8922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:38.831617 8922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:38.831670 8922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:38.831749 8922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:38.836875 8922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:38.837803 8922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:38.838752 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:38.848356 8927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.848400 8930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.848361 8928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.848634 8922 server_base.cc:1034] running on GCE node
I20260927 02:44:38.849015 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:38.849640 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:38.850910 8922 hybrid_clock.cc:648] HybridClock initialized: now 1790477078850894 us; error 35 us; skew 500 ppm
Sep 27 02:44:38 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:38.853714 8922 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:38.854604 8922 webserver.cc:460] Webserver started at http://127.0.98.3:40825/ using document root <none> and password file <none>
I20260927 02:44:38.855185 8922 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:38.855355 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:38.855599 8922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:38.857595 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "6bbbca4fe35740c89d0f419cc5d451e5"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "b777d56abffeb87e6dcec7c799b5254f"
server_key_iv: "078edf1823cc81d31961c4a7bab493da"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.858142 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6bbbca4fe35740c89d0f419cc5d451e5"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "b777d56abffeb87e6dcec7c799b5254f"
server_key_iv: "078edf1823cc81d31961c4a7bab493da"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.861668 8922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:38.864249 8937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.865543 8922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:38.865680 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "6bbbca4fe35740c89d0f419cc5d451e5"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "b777d56abffeb87e6dcec7c799b5254f"
server_key_iv: "078edf1823cc81d31961c4a7bab493da"
server_key_version: "encryptionkey@0"
I20260927 02:44:38.865787 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.895205 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.898553 8922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.898756 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.899343 8922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:38.900223 8922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:38.900291 8922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.900357 8922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:38.900406 8922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.909719 8922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37395
I20260927 02:44:38.909794 9050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37395 every 8 connection(s)
I20260927 02:44:38.910633 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:38.915416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 8922
I20260927 02:44:38.915512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:38.915827 392 external_mini_cluster.cc:1428] Setting key 9d5dff4095d4925447e4ededb39f0f65
Sep 27 02:44:38 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:38.924986 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.912448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35287 (local address 127.0.98.62:35485)
0927 02:44:38.912611 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.912616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.913451 (+ 835us) server_negotiation.cc:407] Connection header received
0927 02:44:38.914848 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.914851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.914899 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.914979 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.916385 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.917185 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.918660 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.918900 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.918987 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.921405 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.921408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.921410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.921434 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.922900 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.923392 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.923395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.923396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.923472 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.923871 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.923936 (+ 65us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.923940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.924138 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.924237 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.924716 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.924865 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:44:38.925750 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.912813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:35287)
0927 02:44:38.913290 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:38.913332 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:38.914624 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:38.915133 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:38.915140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:38.915544 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:38.916206 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.916227 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.917342 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.917346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.918548 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.918558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.919034 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.919037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.919179 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.919716 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:38.919731 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:38.921249 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:38.923025 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.923029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.923040 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.923279 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.923557 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.923559 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.923560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.923693 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.924263 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:38.924268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:38.924494 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 02:44:38.924648 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 02:44:38.924892 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":118,"threads_started":1}
I20260927 02:44:38.926857 9051 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260927 02:44:38.927085 9051 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.927558 9051 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260927 02:44:38.928676 8592 ts_manager.cc:194] Registered new tserver with Master: 6bbbca4fe35740c89d0f419cc5d451e5 (127.0.98.3:37395)
I20260927 02:44:38.929343 8592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35287
I20260927 02:44:38.929976 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:44:39.000299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 8654
I20260927 02:44:39.006124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 8788
I20260927 02:44:39.011204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 8922
I20260927 02:44:39.016419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 8560
I20260927 02:44:39.022318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35485
--webserver_interface=127.0.98.62
--webserver_port=39631
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:39.129315 9060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:39.129590 9060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:39.129670 9060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:39.133635 9060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:39.133711 9060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:39.133750 9060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:39.133807 9060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:39.133852 9060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:39.140375 9060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:39.141402 9060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:39.142385 9060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:39.150630 9066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:39.150648 9065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:39.150650 9068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:39.150689 9060 server_base.cc:1034] running on GCE node
I20260927 02:44:39.151487 9060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:39.152609 9060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:39.153791 9060 hybrid_clock.cc:648] HybridClock initialized: now 1790477079153752 us; error 52 us; skew 500 ppm
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:39.156605 9060 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:39.157444 9060 webserver.cc:460] Webserver started at http://127.0.98.62:39631/ using document root <none> and password file <none>
I20260927 02:44:39.157975 9060 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:39.158075 9060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:39.161751 9060 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:44:39.163765 9075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.165004 9060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 02:44:39.165132 9060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "cfccdd481d6b4d579a524bf2e9474a8d"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "1ed2ff988a3403b413c96a1fa00ce9e2"
server_key_iv: "320b42cb9354d46add3342803d4f5a4e"
server_key_version: "encryptionkey@0"
I20260927 02:44:39.165649 9060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:39.182741 9060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:39.185603 9060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:39.185801 9060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:39.192788 9060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35485
I20260927 02:44:39.192811 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35485 every 8 connection(s)
I20260927 02:44:39.193816 9060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:39.197978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 9060
I20260927 02:44:39.198108 9128 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:44:39.198508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34989
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35485
--webserver_port=34095
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:39.200341 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Bootstrap starting.
I20260927 02:44:39.216742 9128 log.cc:824] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:39.223042 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:39.223392 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Bootstrap complete.
I20260927 02:44:39.227557 9128 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260927 02:44:39.228188 9128 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfccdd481d6b4d579a524bf2e9474a8d, State: Initialized, Role: FOLLOWER
I20260927 02:44:39.228724 9128 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260927 02:44:39.228837 9128 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:39.228924 9128 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:39.229046 9128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:44:39.230815 9128 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260927 02:44:39.231132 9128 leader_election.cc:302] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfccdd481d6b4d579a524bf2e9474a8d; no voters:
I20260927 02:44:39.231460 9128 leader_election.cc:288] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:44:39.231542 9132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:44:39.231750 9132 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [term 2 LEADER]: Becoming Leader. State: Replica: cfccdd481d6b4d579a524bf2e9474a8d, State: Running, Role: LEADER
I20260927 02:44:39.232045 9132 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260927 02:44:39.232636 9128 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:39.232932 9132 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfccdd481d6b4d579a524bf2e9474a8d. Latest consensus state: current_term: 2 leader_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } }
I20260927 02:44:39.233048 9132 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:39.233443 9136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:39.233553 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfccdd481d6b4d579a524bf2e9474a8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } }
I20260927 02:44:39.233711 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:39.236555 9136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:39.237154 9136 catalog_manager.cc:1259] Loaded cluster ID: 5d630c0864274494937df5a30b7ae3ae
I20260927 02:44:39.237210 9136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:39.239485 9136 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:39.240178 9136 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cfccdd481d6b4d579a524bf2e9474a8d: Loaded TSK: 0
I20260927 02:44:39.241318 9136 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:44:39.309756 9130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:39.310040 9130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:39.310115 9130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:39.314188 9130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:39.314262 9130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:39.314352 9130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:39.319379 9130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34095
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:39.320403 9130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:39.321378 9130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:39.331550 9154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:39.331550 9155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:39.331832 9130 server_base.cc:1034] running on GCE node
W20260927 02:44:39.331568 9157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:39.332320 9130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:39.333034 9130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:39.334204 9130 hybrid_clock.cc:648] HybridClock initialized: now 1790477079334189 us; error 39 us; skew 500 ppm
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477079, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:39.337253 9130 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:44:39.338063 9130 webserver.cc:460] Webserver started at http://127.0.98.1:34095/ using document root <none> and password file <none>
I20260927 02:44:39.338608 9130 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:39.338716 9130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:39.342546 9130 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:39.344590 9164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.345814 9130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:39.345909 9130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "96d6d84d0c3f4934b9a2d31b7d9ae7cd"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "f7ac19c65731b2437d6e26f2e720d0fa"
server_key_iv: "f0304cac90e34be90e7e16631f5e4c84"
server_key_version: "encryptionkey@0"
I20260927 02:44:39.346436 9130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:39.362190 9130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:39.365653 9130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:39.365869 9130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:39.366495 9130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:39.367373 9130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:39.367422 9130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.367511 9130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:39.367547 9130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.377204 9130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34989
I20260927 02:44:39.377228 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34989 every 8 connection(s)
I20260927 02:44:39.378234 9130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:39.384821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 9130
I20260927 02:44:39.385277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38327
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35485
--webserver_port=44915
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:39.392704 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.380486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:42513)
0927 02:44:39.380978 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.381021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.382377 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.383536 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.383544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.384073 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.384652 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.384676 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.386132 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.386140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.387285 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.387303 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.388501 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.388504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.388648 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.389307 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.389324 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.391961 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
W20260927 02:44:39.393174 9278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35485 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35485: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260927 02:44:39.393554 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.380305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42513 (local address 127.0.98.62:35485)
0927 02:44:39.380859 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.380884 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.381135 (+ 251us) server_negotiation.cc:407] Connection header received
0927 02:44:39.382807 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.382819 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.383114 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.383460 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.384845 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.385990 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.387532 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.388313 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.388523 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.392208 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42513: BlockingRecv error: recv got EOF from 127.0.98.1:42513 (error 108)
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":134,"threads_started":1}
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:39.399365 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.393628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:48319)
0927 02:44:39.394023 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.394036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.394102 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.394363 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.394365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.394496 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.394585 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.394590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.395684 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.395686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.396316 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.396324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.396977 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.396980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.397047 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.397528 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.397538 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.399280 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330}
W20260927 02:44:39.399689 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.393685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48319 (local address 127.0.98.62:35485)
0927 02:44:39.394099 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.394105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.394124 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:44:39.394184 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.394191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.394246 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.394342 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.394700 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.395555 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.396438 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.396865 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.396971 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.399417 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48319: BlockingRecv error: recv got EOF from 127.0.98.1:48319 (error 108)
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":126,"threads_started":1}
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:39.405071 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.399928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:45255)
0927 02:44:39.400046 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.400060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.400140 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.400346 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.400349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.400527 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.400606 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.400612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.401244 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.401247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.402094 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.402103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.402559 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.402562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.402641 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.403099 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.403118 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.404962 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 02:44:39.405169 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.399968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45255 (local address 127.0.98.62:35485)
0927 02:44:39.400072 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.400075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.400103 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:44:39.400216 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.400218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.400251 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.400314 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.400701 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.401150 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.402225 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.402446 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.402545 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.405066 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45255: BlockingRecv error: recv got EOF from 127.0.98.1:45255 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:44:39.405452 9278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:44:39.495584 9282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:39.495810 9282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:39.495851 9282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:39.499711 9282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:39.499759 9282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:39.499814 9282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:39.504719 9282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44915
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:39.505559 9282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:39.506435 9282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:39.515374 9288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:39.515424 9291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:39.515432 9289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:39.515642 9282 server_base.cc:1034] running on GCE node
I20260927 02:44:39.516053 9282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:39.516665 9282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:39.517840 9282 hybrid_clock.cc:648] HybridClock initialized: now 1790477079517823 us; error 30 us; skew 500 ppm
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477079, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:39.520501 9282 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:44:39.521296 9282 webserver.cc:460] Webserver started at http://127.0.98.2:44915/ using document root <none> and password file <none>
I20260927 02:44:39.521804 9282 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:39.521874 9282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:39.525377 9282 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:39.527323 9298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.528504 9282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:39.528617 9282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "c4968dd4ea264483ae0875a1d9239cdb"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "6340a17c853f6ca4ba370d8497eb551b"
server_key_iv: "30404ceab5a55f9afe7f669fd19cd1b5"
server_key_version: "encryptionkey@0"
I20260927 02:44:39.529088 9282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:39.547346 9282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:39.550081 9282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:39.550261 9282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:39.550808 9282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:39.551586 9282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:39.551637 9282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.551702 9282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:39.551740 9282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.560671 9282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38327
I20260927 02:44:39.560698 9411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38327 every 8 connection(s)
I20260927 02:44:39.561550 9282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:39.571595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 9282
I20260927 02:44:39.572118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37395
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35485
--webserver_port=40825
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:39.572337 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.563409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47627 (local address 127.0.98.62:35485)
0927 02:44:39.563581 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.563586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.564337 (+ 751us) server_negotiation.cc:407] Connection header received
0927 02:44:39.565660 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.565663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.565721 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.565822 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.566898 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.567414 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.568554 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.568777 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.568897 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.572182 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47627: BlockingRecv error: recv got EOF from 127.0.98.2:47627 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:44:39.572878 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.563782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:47627)
0927 02:44:39.564200 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.564238 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.565456 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.565989 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.565996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.566316 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.566748 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.566761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.567559 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.567564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.568430 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.568439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.568921 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.568926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.569111 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.569673 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.569688 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.572021 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
W20260927 02:44:39.573334 9412 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35485 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35485: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:39.578693 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.573767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:58023)
0927 02:44:39.573875 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.573891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.573975 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.574196 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.574199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.574348 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.574430 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.574435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.575128 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.575134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.575665 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.575672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.576186 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.576189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.576282 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.576757 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.576768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.578524 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 02:44:39.578708 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.573878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58023 (local address 127.0.98.62:35485)
0927 02:44:39.573981 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.573985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.574000 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:44:39.574049 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.574052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.574096 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.574182 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.574541 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.575024 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.575781 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.576077 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.576186 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.578627 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58023: BlockingRecv error: recv got EOF from 127.0.98.2:58023 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:39.583979 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.579294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:44191)
0927 02:44:39.579403 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.579420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.579490 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.579701 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.579704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.579844 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.579988 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.579994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.580676 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.580679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.581191 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.581198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.581618 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.581622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.581696 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.582157 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.582175 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.583809 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 02:44:39.584029 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.579385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44191 (local address 127.0.98.62:35485)
0927 02:44:39.579494 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.579499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.579513 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:39.579565 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.579570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.579608 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.579688 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.580094 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.580563 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.581302 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.581513 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.581619 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.583931 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44191: BlockingRecv error: recv got EOF from 127.0.98.2:44191 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 02:44:39.584299 9412 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:44:39.681792 9415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:39.682040 9415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:39.682082 9415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:39.686054 9415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:39.686107 9415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:39.686161 9415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:39.691331 9415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40825
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:39.692344 9415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:39.693260 9415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:39.702450 9421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:39.702510 9420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:39.702453 9423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:39.702828 9415 server_base.cc:1034] running on GCE node
I20260927 02:44:39.703233 9415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:39.703953 9415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:39.705123 9415 hybrid_clock.cc:648] HybridClock initialized: now 1790477079705093 us; error 42 us; skew 500 ppm
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477079, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:39.708122 9415 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:44:39.709004 9415 webserver.cc:460] Webserver started at http://127.0.98.3:40825/ using document root <none> and password file <none>
I20260927 02:44:39.709542 9415 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:39.709621 9415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:39.713392 9415 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:44:39.715420 9430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.716612 9415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:44:39.716747 9415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "6bbbca4fe35740c89d0f419cc5d451e5"
format_stamp: "Formatted at 2026-09-27 02:44:38 on dist-test-slave-gkrl"
server_key: "b777d56abffeb87e6dcec7c799b5254f"
server_key_iv: "078edf1823cc81d31961c4a7bab493da"
server_key_version: "encryptionkey@0"
I20260927 02:44:39.717314 9415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:39.730225 9415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:39.733768 9415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:39.734033 9415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:39.734671 9415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:39.735474 9415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:39.735519 9415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.735584 9415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:39.735616 9415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:39.744395 9415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37395
I20260927 02:44:39.744422 9543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37395 every 8 connection(s)
I20260927 02:44:39.745352 9415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:39.748284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 9415
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:44:39.757349 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.747170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41001 (local address 127.0.98.62:35485)
0927 02:44:39.747309 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.747312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.748359 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 02:44:39.750017 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.750020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.750066 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.750138 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.751537 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.752090 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.753443 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.753649 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.753753 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.757260 (+ 3507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41001: BlockingRecv error: recv got EOF from 127.0.98.3:41001 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:44:39.758185 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.747663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:41001)
0927 02:44:39.748174 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.748233 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.749806 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.750316 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.750323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.750782 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.751402 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.751422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.752205 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.752208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.753324 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.753333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.753790 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.753795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.753973 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.754452 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.754474 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.757171 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":150,"threads_started":1}
I20260927 02:44:39.758213 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.749372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.98.62:35485)
0927 02:44:39.749481 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.749493 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.749512 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:44:39.749550 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.749552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.749588 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.749690 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.750460 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.751305 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.752310 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.752636 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.752749 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.754221 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:39.754228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:39.754240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:39.754263 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:39.756056 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:39.756503 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:39.756511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:39.756520 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:39.756670 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:39.756960 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:39.756963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:39.756964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:39.757381 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:39.757567 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:39.757872 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 02:44:39.758080 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 02:44:39.758700 9544 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35485 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35485: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:39.764153 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.759153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:40203)
0927 02:44:39.759251 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.759266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.759342 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.759563 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.759565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.759718 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.759816 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.759822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.760729 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.760732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.761394 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.761404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.761844 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.761848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.761922 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.762389 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.762402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.764071 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:44:39.764369 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.759199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40203 (local address 127.0.98.62:35485)
0927 02:44:39.759298 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.759308 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.759320 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:44:39.759430 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.759432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.759461 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.759521 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.760002 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.760627 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.761501 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.761713 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.761799 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.764265 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40203: BlockingRecv error: recv got EOF from 127.0.98.3:40203 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:39 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:39.769037 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.764631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:42471)
0927 02:44:39.764733 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.764747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.764818 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.765031 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.765033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.765151 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:39.765223 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.765228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.765893 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.765896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.766387 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.766393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.766779 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.766781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.766867 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.767223 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:39.767236 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:39.768933 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:44:39.769215 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:39.764701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42471 (local address 127.0.98.62:35485)
0927 02:44:39.764828 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.764834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.764850 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:39.764878 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.764880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.764908 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.764980 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.765326 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.765765 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.766494 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.766685 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.766764 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.769075 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42471: BlockingRecv error: recv got EOF from 127.0.98.3:42471 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 02:44:39.769243 9544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:44:40 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:40 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:40.411581 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:40.406100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:54087)
0927 02:44:40.406376 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:40.406402 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:40.406502 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:40.406797 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:40.406800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:40.407054 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:40.407173 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:40.407179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.407990 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.407993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:40.408611 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:40.408620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.409076 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.409079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:40.409158 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:40.409708 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:40.409720 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:40.411469 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
W20260927 02:44:40.411757 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:40.406157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54087 (local address 127.0.98.62:35485)
0927 02:44:40.406445 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:40.406448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:40.406460 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:44:40.406611 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:40.406614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:40.406672 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:40.406761 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:40.407276 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.407808 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.408722 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.408960 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.409039 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:40.411589 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54087: BlockingRecv error: recv got EOF from 127.0.98.1:54087 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
Sep 27 02:44:40 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:40 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:40.590969 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:40.585062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:33379)
0927 02:44:40.585343 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:40.585358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:40.585493 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:40.585751 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:40.585753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:40.585939 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:40.586071 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:40.586077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.586809 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.586812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:40.587586 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:40.587597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.588187 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.588190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:40.588313 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:40.588891 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:40.588907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:40.590838 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
W20260927 02:44:40.591130 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:40.585127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33379 (local address 127.0.98.62:35485)
0927 02:44:40.585271 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:40.585275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:40.585456 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:44:40.585581 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:40.585583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:40.585626 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:40.585706 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:40.586186 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.586706 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.587695 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.588055 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.588174 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:40.590975 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33379: BlockingRecv error: recv got EOF from 127.0.98.2:33379 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 02:44:40 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:40 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:40.775622 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:40.769785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:41773)
0927 02:44:40.770170 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:40.770189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:40.770325 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:40.770619 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:40.770624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:40.770874 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:40.771062 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:40.771075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.772133 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.772136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:40.772704 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:40.772711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.773145 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.773148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:40.773215 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:40.773691 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:40.773703 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:40.775503 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
W20260927 02:44:40.775741 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:40.769965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41773 (local address 127.0.98.62:35485)
0927 02:44:40.770096 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:40.770101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:40.770360 (+ 259us) server_negotiation.cc:407] Connection header received
0927 02:44:40.770419 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:40.770423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:40.770481 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:40.770570 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:40.771322 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.772041 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.772811 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:40.773047 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:40.773129 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:40.775629 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41773: BlockingRecv error: recv got EOF from 127.0.98.3:41773 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 02:44:41 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:41 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:41.417912 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:41.412359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:41603)
0927 02:44:41.412628 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:41.412641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:41.412733 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:41.412965 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:41.412967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:41.413139 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:41.413231 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:41.413236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.414022 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.414024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:41.414817 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:41.414829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.415287 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.415290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:41.415353 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:41.415897 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:41.415911 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:41.417720 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
W20260927 02:44:41.418015 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:41.412416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41603 (local address 127.0.98.62:35485)
0927 02:44:41.412666 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:41.412670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:41.412692 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:44:41.412804 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:41.412806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:41.412848 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:41.412930 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:41.413332 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.413919 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.414953 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.415176 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.415262 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:41.417886 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41603: BlockingRecv error: recv got EOF from 127.0.98.1:41603 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":65,"threads_started":1}
Sep 27 02:44:41 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:41 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:41.597092 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:41.591741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:45315)
0927 02:44:41.592041 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:41.592054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:41.592148 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:41.592376 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:41.592378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:41.592551 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:41.592646 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:41.592651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.593414 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.593416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:41.593993 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:41.594001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.594477 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.594481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:41.594565 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:41.595050 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:41.595061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:41.596913 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":115,"threads_started":1}
W20260927 02:44:41.597127 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:41.591824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45315 (local address 127.0.98.62:35485)
0927 02:44:41.592038 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:41.592041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:41.592102 (+ 61us) server_negotiation.cc:407] Connection header received
0927 02:44:41.592211 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:41.592213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:41.592253 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:41.592322 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:41.592779 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.593304 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.594136 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.594361 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.594473 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:41.597029 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45315: BlockingRecv error: recv got EOF from 127.0.98.2:45315 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 27 02:44:41 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:41 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:41.782258 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:41.776411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:53689)
0927 02:44:41.776835 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:41.776852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:41.776998 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:41.777267 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:41.777270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:41.777476 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:41.777568 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:41.777574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.778566 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.778569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:41.779264 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:41.779273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.779708 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.779712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:41.779776 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:41.780337 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:41.780349 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:41.782131 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":123,"threads_started":1}
W20260927 02:44:41.782423 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:41.776574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53689 (local address 127.0.98.62:35485)
0927 02:44:41.776812 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:41.776817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:41.776935 (+ 118us) server_negotiation.cc:407] Connection header received
0927 02:44:41.777096 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:41.777099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:41.777155 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:41.777242 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:41.777711 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.778453 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.779363 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:41.779580 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:41.779659 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:41.782265 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53689: BlockingRecv error: recv got EOF from 127.0.98.3:53689 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Sep 27 02:44:42 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:42 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:42.428248 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.418713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:41803)
0927 02:44:42.418992 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.419005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.419101 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.419446 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.419448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.419626 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.419723 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.419729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.420658 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.420661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.421211 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.421220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.421800 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.421804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.421873 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.426135 (+ 4262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.426147 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.428112 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
W20260927 02:44:42.428443 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:42.418839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41803 (local address 127.0.98.62:35485)
0927 02:44:42.419210 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.419214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.419228 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:42.419264 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.419266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.419308 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.419409 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.419854 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.420518 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.421327 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.421637 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.421759 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.428263 (+ 6504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41803: BlockingRecv error: recv got EOF from 127.0.98.1:41803 (error 108)
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":224,"threads_started":1}
Sep 27 02:44:42 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:42 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:42.603174 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.597947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:40195)
0927 02:44:42.598220 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.598234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.598377 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.598688 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.598693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.598924 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.599022 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.599028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.599752 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.599755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.600357 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.600365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.600781 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.600785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.600855 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.601329 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.601340 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.603066 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
W20260927 02:44:42.603350 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:42.598049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40195 (local address 127.0.98.62:35485)
0927 02:44:42.598206 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.598211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.598326 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:44:42.598505 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.598508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.598561 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.598673 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.599128 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.599647 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.600464 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.600676 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.600755 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.603194 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40195: BlockingRecv error: recv got EOF from 127.0.98.2:40195 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 02:44:42 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:42 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:42.788772 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.783002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:54443)
0927 02:44:42.783270 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.783284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.783376 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.783601 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.783604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.783778 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.783958 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.783964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.784726 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.784733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.785308 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.785316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.785766 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.785769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.785865 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.786392 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.786405 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.788572 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
W20260927 02:44:42.788810 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:42.783039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54443 (local address 127.0.98.62:35485)
0927 02:44:42.783173 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.783176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.783349 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:44:42.783452 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.783454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.783497 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.783565 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.784085 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.784616 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.785427 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.785653 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.785764 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.788699 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54443: BlockingRecv error: recv got EOF from 127.0.98.3:54443 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 02:44:43 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:43 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:43.434840 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:43.429039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:36159)
0927 02:44:43.429304 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:43.429316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:43.429420 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:43.429712 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:43.429714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:43.429889 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:43.429981 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:43.429985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.430725 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.430728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:43.431306 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:43.431315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.432028 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.432031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:43.432101 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:43.432573 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:43.432584 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:43.434715 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260927 02:44:43.435004 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:43.429103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36159 (local address 127.0.98.62:35485)
0927 02:44:43.429354 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:43.429357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:43.429375 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:44:43.429545 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:43.429548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:43.429593 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:43.429683 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:43.430081 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.430613 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.431565 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.431794 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.432012 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:43.434851 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36159: BlockingRecv error: recv got EOF from 127.0.98.1:36159 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
Sep 27 02:44:43 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:43 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:43.609688 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:43.604088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:57687)
0927 02:44:43.604364 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:43.604378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:43.604479 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:43.604713 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:43.604716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:43.604910 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:43.605010 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:43.605016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.605825 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.605828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:43.606455 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:43.606463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.606942 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.606945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:43.607016 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:43.607541 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:43.607554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:43.609537 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
W20260927 02:44:43.609850 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:43.604108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57687 (local address 127.0.98.62:35485)
0927 02:44:43.604253 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:43.604256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:43.604448 (+ 192us) server_negotiation.cc:407] Connection header received
0927 02:44:43.604558 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:43.604560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:43.604601 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:43.604670 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:43.605143 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.605699 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.606577 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.606808 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.606919 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:43.609737 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57687: BlockingRecv error: recv got EOF from 127.0.98.2:57687 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 02:44:43 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:43 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:43.794997 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:43.789662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:58161)
0927 02:44:43.789954 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:43.789967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:43.790064 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:43.790284 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:43.790286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:43.790463 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:43.790555 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:43.790567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.791296 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.791298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:43.791830 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:43.791837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.792290 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.792292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:43.792400 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:43.793006 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:43.793028 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:43.794881 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
W20260927 02:44:43.795123 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:43.789710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58161 (local address 127.0.98.62:35485)
0927 02:44:43.789906 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:43.789912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:43.790041 (+ 129us) server_negotiation.cc:407] Connection header received
0927 02:44:43.790144 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:43.790146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:43.790189 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:43.790249 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:43.790659 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.791205 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.791956 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:43.792194 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:43.792291 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:43.795007 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58161: BlockingRecv error: recv got EOF from 127.0.98.3:58161 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 27 02:44:44 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:44 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:44.441552 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:44.435835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:38079)
0927 02:44:44.436264 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:44.436282 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:44.436426 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:44.436686 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:44.436688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:44.436875 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:44.436972 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:44.436977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.437704 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.437707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:44.438345 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:44.438354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.438791 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.438794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:44.438871 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:44.439459 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:44.439471 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:44.441435 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":83,"threads_started":1}
W20260927 02:44:44.441787 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:44.435976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38079 (local address 127.0.98.62:35485)
0927 02:44:44.436221 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:44.436250 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:44.436393 (+ 143us) server_negotiation.cc:407] Connection header received
0927 02:44:44.436524 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:44.436526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:44.436570 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:44.436648 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:44.437080 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.437599 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.438468 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.438687 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.438816 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:44.441650 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38079: BlockingRecv error: recv got EOF from 127.0.98.1:38079 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
Sep 27 02:44:44 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:44 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:44.615818 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:44.610738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:39395)
0927 02:44:44.610995 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:44.611009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:44.611095 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:44.611306 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:44.611308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:44.611481 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:44.611570 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:44.611575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.612310 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.612313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:44.612880 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:44.612887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.613289 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.613291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:44.613361 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:44.613830 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:44.613840 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:44.615699 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
W20260927 02:44:44.616034 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:44.610854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39395 (local address 127.0.98.62:35485)
0927 02:44:44.611008 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:44.611012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:44.611093 (+ 81us) server_negotiation.cc:407] Connection header received
0927 02:44:44.611160 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:44.611162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:44.611206 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:44.611277 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:44.611666 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.612217 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.612988 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.613196 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.613272 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:44.615843 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39395: BlockingRecv error: recv got EOF from 127.0.98.2:39395 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 02:44:44 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:44 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:44.801244 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:44.795975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:52385)
0927 02:44:44.796214 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:44.796227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:44.796328 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:44.796512 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:44.796514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:44.796683 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:44.796775 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:44.796779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.797519 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.797522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:44.798118 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:44.798127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.798594 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.798598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:44.798680 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:44.799274 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:44.799286 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:44.801099 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
W20260927 02:44:44.801316 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:44.796006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52385 (local address 127.0.98.62:35485)
0927 02:44:44.796130 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:44.796134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:44.796301 (+ 167us) server_negotiation.cc:407] Connection header received
0927 02:44:44.796393 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:44.796395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:44.796435 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:44.796504 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:44.796895 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.797399 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.798221 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.798436 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.798512 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:44.801218 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52385: BlockingRecv error: recv got EOF from 127.0.98.3:52385 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 02:44:45 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:45 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:45.448348 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:45.442382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:59519)
0927 02:44:45.442659 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:45.442677 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:45.442794 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:45.443119 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:45.443122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:45.443369 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:45.443487 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:45.443493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.444453 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.444456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:45.445005 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:45.445013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.445500 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.445503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:45.445571 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:45.446169 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:45.446180 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:45.448183 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
W20260927 02:44:45.448448 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:45.442446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59519 (local address 127.0.98.62:35485)
0927 02:44:45.442697 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:45.442700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:45.442757 (+ 57us) server_negotiation.cc:407] Connection header received
0927 02:44:45.442909 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:45.442912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:45.442973 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:45.443066 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:45.443610 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.444351 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.445140 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.445378 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.445466 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:45.448302 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59519: BlockingRecv error: recv got EOF from 127.0.98.1:59519 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
Sep 27 02:44:45 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:45 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:45.622607 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:45.616825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:34247)
0927 02:44:45.617098 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:45.617112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:45.617201 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:45.617474 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:45.617476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:45.617656 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:45.617804 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:45.617810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.618571 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.618573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:45.619182 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:45.619191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.619646 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.619649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:45.619715 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:45.620409 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:45.620421 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:45.622496 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
W20260927 02:44:45.622766 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:45.616976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34247 (local address 127.0.98.62:35485)
0927 02:44:45.617139 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:45.617143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:45.617157 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:44:45.617280 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:45.617283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:45.617335 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:45.617397 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:45.617937 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.618466 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.619304 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.619525 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.619606 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:45.622643 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34247: BlockingRecv error: recv got EOF from 127.0.98.2:34247 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 02:44:45 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:45 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:45.807667 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:45.802099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:56255)
0927 02:44:45.802416 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:45.802429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:45.802575 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:45.802952 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:45.802955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:45.803158 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:45.803248 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:45.803253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.804093 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.804098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:45.804654 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:45.804661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.805120 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.805123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:45.805208 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:45.805700 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:45.805711 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:45.807558 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
W20260927 02:44:45.807910 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:45.802158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56255 (local address 127.0.98.62:35485)
0927 02:44:45.802354 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:45.802358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:45.802512 (+ 154us) server_negotiation.cc:407] Connection header received
0927 02:44:45.802711 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:45.802714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:45.802823 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:45.802925 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:45.803380 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.803989 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.804780 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:45.805007 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:45.805117 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:45.807746 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56255: BlockingRecv error: recv got EOF from 127.0.98.3:56255 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 02:44:46 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:46 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:46.455309 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.449186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:49591)
0927 02:44:46.449509 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.449522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.449616 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.449847 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.449849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.450021 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:46.450116 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.450121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.451158 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.451162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.452112 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.452124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.452610 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.452612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.452668 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.453372 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:46.453384 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:46.455192 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
W20260927 02:44:46.455489 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:46.449284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49591 (local address 127.0.98.62:35485)
0927 02:44:46.449529 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.449532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.449570 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:44:46.449693 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.449695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.449734 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.449810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:46.450215 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.451016 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.452282 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.452510 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.452596 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.455339 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49591: BlockingRecv error: recv got EOF from 127.0.98.1:49591 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
Sep 27 02:44:46 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:46 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:46.629076 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.623466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:57845)
0927 02:44:46.623812 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.623827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.623959 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.624185 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.624196 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.624374 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:46.624470 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.624475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.625290 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.625294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.625894 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.625903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.626376 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.626379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.626445 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.627064 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:46.627079 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:46.628962 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":94,"threads_started":1}
W20260927 02:44:46.629366 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:46.623539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57845 (local address 127.0.98.62:35485)
0927 02:44:46.623694 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.623698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.623939 (+ 241us) server_negotiation.cc:407] Connection header received
0927 02:44:46.624028 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.624031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.624076 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.624148 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:46.624598 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.625169 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.626024 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.626265 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.626347 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.629136 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57845: BlockingRecv error: recv got EOF from 127.0.98.2:57845 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 02:44:46 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:46 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:46.813786 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.808401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:54791)
0927 02:44:46.808770 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.808788 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.808934 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.809200 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.809203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.809410 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:46.809498 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.809504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.810264 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.810268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.810819 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.810826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.811268 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.811272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.811357 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.811840 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:46.811851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:46.813679 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":100,"threads_started":1}
W20260927 02:44:46.814055 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:46.808521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54791 (local address 127.0.98.62:35485)
0927 02:44:46.808694 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.808707 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.808875 (+ 168us) server_negotiation.cc:407] Connection header received
0927 02:44:46.809050 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.809053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.809096 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.809180 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:46.809615 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.810142 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.810937 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.811156 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.811266 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.813948 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54791: BlockingRecv error: recv got EOF from 127.0.98.3:54791 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 02:44:47 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:47 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:47.464507 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.456029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:56905)
0927 02:44:47.456299 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.456316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.456424 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.456651 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.456653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.456828 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:47.456922 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.456927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.457639 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.457641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.458227 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.458242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.458668 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.458670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.458730 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.462175 (+ 3445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:47.462187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:47.464335 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":62,"threads_started":1}
W20260927 02:44:47.464576 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:47.456184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56905 (local address 127.0.98.62:35485)
0927 02:44:47.456414 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.456418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.456431 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:47.456500 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.456502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.456541 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.456616 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.457022 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.457548 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.458348 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.458568 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.458647 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.464457 (+ 5810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56905: BlockingRecv error: recv got EOF from 127.0.98.1:56905 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
Sep 27 02:44:47 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:47 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:47.636061 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.629817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:54683)
0927 02:44:47.630099 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.630122 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.630243 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.630506 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.630508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.630712 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:47.630799 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.630804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.631992 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.631997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.632790 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.632799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.633381 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.633384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.633456 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.633987 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:47.634000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:47.635940 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
W20260927 02:44:47.636189 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:47.629899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54683 (local address 127.0.98.62:35485)
0927 02:44:47.630058 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.630063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.630189 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:44:47.630350 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.630353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.630405 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.630488 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.630930 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.631870 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.632995 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.633263 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.633389 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.636050 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54683: BlockingRecv error: recv got EOF from 127.0.98.2:54683 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 02:44:47 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:47 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:47.820250 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.814517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:32903)
0927 02:44:47.814882 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.814894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.815028 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.815295 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.815298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.815502 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:47.815590 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.815596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.816415 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.816421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.816974 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.816981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.817430 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.817434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.817518 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.818132 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:47.818144 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:47.820087 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":113,"threads_started":1}
W20260927 02:44:47.820322 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:47.814591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32903 (local address 127.0.98.62:35485)
0927 02:44:47.814753 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.814760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.814973 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:44:47.815129 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.815133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.815185 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.815274 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.815709 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.816302 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.817094 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.817313 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.817429 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.820214 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32903: BlockingRecv error: recv got EOF from 127.0.98.3:32903 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 02:44:48 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:48 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:48.470849 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.465232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:60529)
0927 02:44:48.465497 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:48.465515 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:48.465648 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:48.465890 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:48.465892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:48.466064 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:48.466158 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.466163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.466893 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.466896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.467459 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.467466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.467914 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.467917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.467989 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.468473 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:48.468484 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:48.470744 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
W20260927 02:44:48.471040 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:48.465351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60529 (local address 127.0.98.62:35485)
0927 02:44:48.465684 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:48.465688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:48.465728 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:44:48.465762 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:48.465764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:48.465805 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:48.465874 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:48.466255 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.466792 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.467558 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.467769 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.467845 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.470897 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60529: BlockingRecv error: recv got EOF from 127.0.98.1:60529 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
Sep 27 02:44:48 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:48 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:48.642496 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.636806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:58859)
0927 02:44:48.637107 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:48.637120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:48.637219 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:48.637441 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:48.637444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:48.637636 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:48.637726 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.637731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.638433 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.638436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.639244 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.639256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.639964 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.639970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.640063 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.640588 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:48.640602 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:48.642389 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
W20260927 02:44:48.642663 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:48.636921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58859 (local address 127.0.98.62:35485)
0927 02:44:48.637105 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:48.637108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:48.637207 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:44:48.637297 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:48.637299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:48.637341 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:48.637401 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:48.637827 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.638331 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.639378 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.639815 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.639969 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.642514 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58859: BlockingRecv error: recv got EOF from 127.0.98.2:58859 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 27 02:44:48 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:48 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:48.826519 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.821070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:42609)
0927 02:44:48.821356 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:48.821370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:48.821468 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:48.821696 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:48.821698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:48.821953 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:48.822050 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.822056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.822845 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.822847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.823397 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.823404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.823847 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.823850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.823940 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.824489 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:48.824501 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:48.826353 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
W20260927 02:44:48.826578 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:48.821142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42609 (local address 127.0.98.62:35485)
0927 02:44:48.821283 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:48.821301 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:48.821446 (+ 145us) server_negotiation.cc:407] Connection header received
0927 02:44:48.821549 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:48.821551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:48.821596 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:48.821661 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:48.822146 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.822747 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.823525 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.823737 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.823826 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.826475 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42609: BlockingRecv error: recv got EOF from 127.0.98.3:42609 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 02:44:49 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:49 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:49.477566 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:49.471655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:35539)
0927 02:44:49.472065 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:49.472082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:49.472202 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:49.472531 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:49.472536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:49.472767 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:49.472904 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:49.472926 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.473766 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.473768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:49.474322 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:49.474329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.474843 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.474846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:49.474909 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:49.475462 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:49.475476 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:49.477394 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":78,"threads_started":1}
W20260927 02:44:49.477655 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:49.471768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35539 (local address 127.0.98.62:35485)
0927 02:44:49.472033 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:49.472036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:49.472174 (+ 138us) server_negotiation.cc:407] Connection header received
0927 02:44:49.472303 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:49.472308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:49.472367 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:49.472474 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:49.473061 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.473672 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.474457 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.474700 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.474835 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:49.477522 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35539: BlockingRecv error: recv got EOF from 127.0.98.1:35539 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":58,"threads_started":1}
Sep 27 02:44:49 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:49 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:49.648713 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:49.643243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:38277)
0927 02:44:49.643534 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:49.643547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:49.643673 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:49.644005 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:49.644008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:49.644207 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:49.644297 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:49.644303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.645054 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.645060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:49.645628 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:49.645636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.646076 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.646080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:49.646164 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:49.646648 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:49.646660 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:49.648547 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
W20260927 02:44:49.648818 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:49.643302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38277 (local address 127.0.98.62:35485)
0927 02:44:49.643496 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:49.643501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:49.643619 (+ 118us) server_negotiation.cc:407] Connection header received
0927 02:44:49.643776 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:49.643780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:49.643830 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:49.643981 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:49.644419 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.644950 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.645744 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.645971 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.646076 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:49.648661 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38277: BlockingRecv error: recv got EOF from 127.0.98.2:38277 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 02:44:49 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:49 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:49.833066 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:49.827272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:43353)
0927 02:44:49.827722 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:49.827739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:49.827866 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:49.828147 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:49.828150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:49.828381 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:49.828472 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:49.828477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.829257 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.829263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:49.829825 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:49.829832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.830324 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.830327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:49.830421 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:49.830918 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:49.830930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:49.832896 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":141,"threads_started":1}
W20260927 02:44:49.833104 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:49.827454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43353 (local address 127.0.98.62:35485)
0927 02:44:49.827710 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:49.827714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:49.827821 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:44:49.827979 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:49.827983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:49.828042 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:49.828127 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:49.828582 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.829146 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.829946 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:49.830164 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:49.830315 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:49.833008 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43353: BlockingRecv error: recv got EOF from 127.0.98.3:43353 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
Sep 27 02:44:50 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:50 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:44:50.484010 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:50.478520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42355 (local address 127.0.98.62:35485)
0927 02:44:50.478766 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:50.478771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:50.478811 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:44:50.478843 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:50.478845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:50.478886 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:50.478956 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:50.479398 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.479940 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.480723 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.480946 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.481028 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:50.483868 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42355: BlockingRecv error: recv got EOF from 127.0.98.1:42355 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260927 02:44:50.484006 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:50.478342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:42355)
0927 02:44:50.478644 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:50.478664 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:50.478761 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:50.478992 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:50.478994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:50.479170 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:50.479263 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:50.479268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.480050 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.480053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:50.480609 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:50.480617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.481048 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.481051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:50.481110 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:50.481668 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:50.481679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:50.483796 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
Sep 27 02:44:50 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:50 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:50.655827 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:50.649698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:46691)
0927 02:44:50.649991 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:50.650015 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:50.650162 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:50.650453 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:50.650456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:50.650686 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:50.650876 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:50.650889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.651719 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.651724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:50.652462 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:50.652471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.652972 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.652977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:50.653059 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:50.653649 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:50.653677 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:50.655664 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
W20260927 02:44:50.656004 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:50.649794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46691 (local address 127.0.98.62:35485)
0927 02:44:50.649958 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:50.649963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:50.650096 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:44:50.650279 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:50.650282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:50.650338 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:50.650432 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:50.651003 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.651592 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.652601 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.652852 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.652983 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:50.655826 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46691: BlockingRecv error: recv got EOF from 127.0.98.2:46691 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 02:44:50 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:50 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:50.839716 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:50.833907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:42315)
0927 02:44:50.834244 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:50.834261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:50.834396 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:50.834704 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:50.834709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:50.834980 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:50.835108 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:50.835116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.835837 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.835839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:50.836476 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:50.836485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.836967 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.836970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:50.837050 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:50.837631 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:50.837643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:50.839609 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":92,"threads_started":1}
W20260927 02:44:50.839938 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:50.833945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42315 (local address 127.0.98.62:35485)
0927 02:44:50.834078 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:50.834083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:50.834347 (+ 264us) server_negotiation.cc:407] Connection header received
0927 02:44:50.834505 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:50.834509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:50.834568 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:50.834654 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:50.835208 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.835710 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.836577 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:50.836792 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:50.836902 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:50.839773 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42315: BlockingRecv error: recv got EOF from 127.0.98.3:42315 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 02:44:51 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:51 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:51.490535 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:51.484778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:43655)
0927 02:44:51.485047 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:51.485060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:51.485146 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:51.485371 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:51.485373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:51.485557 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:51.485653 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:51.485658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.486511 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.486514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:51.487076 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:51.487085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.487654 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.487658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:51.487729 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:51.488431 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:51.488443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:51.490421 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
W20260927 02:44:51.490823 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:51.484923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43655 (local address 127.0.98.62:35485)
0927 02:44:51.485153 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:51.485157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:51.485170 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:51.485207 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:51.485209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:51.485253 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:51.485335 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:51.485879 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.486397 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.487212 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.487483 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.487576 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:51.490612 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43655: BlockingRecv error: recv got EOF from 127.0.98.1:43655 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
Sep 27 02:44:51 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:51 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:51.662482 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:51.656895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:39331)
0927 02:44:51.657180 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:51.657194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:51.657328 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:51.657605 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:51.657607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:51.657778 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:51.657874 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:51.657879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.658651 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.658654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:51.659250 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:51.659258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.659766 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.659770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:51.659844 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:51.660443 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:51.660455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:51.662371 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":111,"threads_started":1}
W20260927 02:44:51.662679 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:51.656957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39331 (local address 127.0.98.62:35485)
0927 02:44:51.657109 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:51.657114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:51.657250 (+ 136us) server_negotiation.cc:407] Connection header received
0927 02:44:51.657424 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:51.657427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:51.657477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:51.657549 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:51.657997 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.658529 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.659392 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.659612 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.659704 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:51.662508 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39331: BlockingRecv error: recv got EOF from 127.0.98.2:39331 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 02:44:51 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:51 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:51.846366 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:51.840555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:42979)
0927 02:44:51.840868 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:51.840882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:51.840980 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:51.841273 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:51.841275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:51.841463 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:51.841563 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:51.841568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.842326 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.842329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:51.843006 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:51.843016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.843429 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.843431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:51.843496 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:51.844167 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:51.844180 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:51.846185 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":122,"threads_started":1}
W20260927 02:44:51.846450 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:51.840616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42979 (local address 127.0.98.62:35485)
0927 02:44:51.840774 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:51.840777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:51.840953 (+ 176us) server_negotiation.cc:407] Connection header received
0927 02:44:51.841098 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:51.841101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:51.841149 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:51.841222 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:51.841676 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.842216 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.843116 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:51.843331 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:51.843404 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:51.846328 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42979: BlockingRecv error: recv got EOF from 127.0.98.3:42979 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 02:44:52 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:52 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:52.499462 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:52.491364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:42297)
0927 02:44:52.491637 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:52.491650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:52.491772 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:52.492011 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:52.492013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:52.492183 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:52.492295 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:52.492300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.493036 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.493039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:52.493616 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:52.493626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.494140 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.494143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:52.494213 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:52.497507 (+ 3294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:52.497519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:52.499351 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
W20260927 02:44:52.499658 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:52.491394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42297 (local address 127.0.98.62:35485)
0927 02:44:52.491659 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:52.491662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:52.491723 (+ 61us) server_negotiation.cc:407] Connection header received
0927 02:44:52.491862 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:52.491864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:52.491923 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:52.492000 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:52.492390 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.492905 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.493791 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.494021 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.494102 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:52.499485 (+ 5383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42297: BlockingRecv error: recv got EOF from 127.0.98.1:42297 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
Sep 27 02:44:52 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:52 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:52.668424 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:52.663233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:46893)
0927 02:44:52.663530 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:52.663542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:52.663634 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:52.663853 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:52.663855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:52.664033 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:52.664123 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:52.664128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.664851 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.664855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:52.665407 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:52.665415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.665883 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.665885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:52.665948 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:52.666490 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:52.666502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:52.668313 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":127,"threads_started":1}
W20260927 02:44:52.668622 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:52.663356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46893 (local address 127.0.98.62:35485)
0927 02:44:52.663509 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:52.663513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:52.663607 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:44:52.663713 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:52.663716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:52.663764 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:52.663842 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:52.664220 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.664723 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.665552 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.665768 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.665862 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:52.668485 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46893: BlockingRecv error: recv got EOF from 127.0.98.2:46893 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 02:44:52 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:52 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:52.852717 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:52.847166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:58671)
0927 02:44:52.847469 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:52.847482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:52.847577 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:52.847863 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:52.847869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:52.848138 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:52.848310 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:52.848319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.849017 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.849020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:52.849662 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:52.849670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.850075 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.850077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:52.850133 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:52.850685 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:52.850743 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:52.852604 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":107,"threads_started":1}
W20260927 02:44:52.852820 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:52.847241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58671 (local address 127.0.98.62:35485)
0927 02:44:52.847389 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:52.847395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:52.847552 (+ 157us) server_negotiation.cc:407] Connection header received
0927 02:44:52.847655 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:52.847657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:52.847700 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:52.847764 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:52.848425 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.848928 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.849757 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:52.849982 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:52.850059 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:52.852733 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58671: BlockingRecv error: recv got EOF from 127.0.98.3:58671 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 02:44:53 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:53 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:53.505352 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:53.500224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:44855)
0927 02:44:53.500505 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:53.500518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:53.500600 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:53.500819 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:53.500821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:53.500992 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:53.501087 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:53.501092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.501826 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.501829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:53.502375 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:53.502383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.502794 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.502822 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:53.502881 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:53.503458 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:53.503470 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:53.505228 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
W20260927 02:44:53.505532 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:53.500301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44855 (local address 127.0.98.62:35485)
0927 02:44:53.500586 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:53.500589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:53.500602 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:44:53.500680 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:53.500683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:53.500723 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:53.500803 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:53.501183 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.501704 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.502481 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.502698 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.502821 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:53.505354 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44855: BlockingRecv error: recv got EOF from 127.0.98.1:44855 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
Sep 27 02:44:53 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:53 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:53.674968 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:53.669298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:43537)
0927 02:44:53.669595 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:53.669611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:53.669721 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:53.670049 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:53.670052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:53.670339 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:53.670434 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:53.670440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.671225 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.671228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:53.671784 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:53.671790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.672338 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.672344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:53.672413 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:53.672996 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:53.673007 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:53.674843 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
W20260927 02:44:53.675146 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:53.669395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43537 (local address 127.0.98.62:35485)
0927 02:44:53.669524 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:53.669529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:53.669698 (+ 169us) server_negotiation.cc:407] Connection header received
0927 02:44:53.669861 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:53.669867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:53.669922 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:53.670010 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:53.670566 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.671126 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.671951 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.672200 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.672286 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:53.674979 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43537: BlockingRecv error: recv got EOF from 127.0.98.2:43537 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 02:44:53 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:53 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:53.859017 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:53.853631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:53751)
0927 02:44:53.853897 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:53.853915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:53.854049 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:53.854346 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:53.854349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:53.854574 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:53.854665 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:53.854671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.855415 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.855420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:53.856009 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:53.856017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.856565 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.856567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:53.856631 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:53.857105 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:53.857116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:53.858901 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":104,"threads_started":1}
W20260927 02:44:53.859143 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:53.853699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53751 (local address 127.0.98.62:35485)
0927 02:44:53.853858 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:53.853862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:53.853992 (+ 130us) server_negotiation.cc:407] Connection header received
0927 02:44:53.854175 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:53.854177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:53.854224 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:53.854322 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:53.854784 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.855305 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.856174 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:53.856458 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:53.856550 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:53.859029 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53751: BlockingRecv error: recv got EOF from 127.0.98.3:53751 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 02:44:54 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:54 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:54.512135 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:54.506251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:43073)
0927 02:44:54.506600 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:54.506618 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:54.506736 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:54.507102 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:54.507110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:54.507355 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:54.507463 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:54.507469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.508323 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.508329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:54.508912 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:54.508920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.509379 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.509382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:54.509470 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:54.510041 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:54.510052 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:54.512024 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":79,"threads_started":1}
W20260927 02:44:54.512327 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:54.506389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43073 (local address 127.0.98.62:35485)
0927 02:44:54.506637 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:54.506640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:54.506671 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:44:54.506901 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:54.506907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:54.506967 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:54.507040 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:54.507586 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.508215 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.509033 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.509264 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.509380 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:54.512159 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43073: BlockingRecv error: recv got EOF from 127.0.98.1:43073 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
Sep 27 02:44:54 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:44:54 dist-test-slave-gkrl krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:44:54.681627 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:54.675807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:45023)
0927 02:44:54.676133 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:54.676149 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:54.676284 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:54.676550 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:54.676553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:54.676756 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:54.676872 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:54.676878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.677636 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.677642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:54.678205 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:54.678213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.678676 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.678680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:54.678764 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:54.679331 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:54.679342 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:54.681517 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35485: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":80,"threads_started":1}
W20260927 02:44:54.681769 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:54.675963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45023 (local address 127.0.98.62:35485)
0927 02:44:54.676110 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:54.676114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:54.676227 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:44:54.676390 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:54.676393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:54.676448 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:54.676530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:54.676989 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.677522 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.678326 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.678565 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.678674 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:54.681647 (+ 2973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45023: BlockingRecv error: recv got EOF from 127.0.98.2:45023 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:44:54.749193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 9130
I20260927 02:44:54.754501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 9282
I20260927 02:44:54.760071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 9415
I20260927 02:44:54.765115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 9060
2026-09-27T02:44:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17913 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T02:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:54Z Disabled control of system clock
I20260927 02:44:54.795186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43795
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:44:54.903097 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:54.903347 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:54.903383 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:54.907343 9610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:54.907418 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:54.907434 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:54.907450 9610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:54.907465 9610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:54.912292 9610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:54.913326 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:54.914695 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:54.923516 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:54.923590 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:54.923514 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:54.923534 9610 server_base.cc:1034] running on GCE node
I20260927 02:44:54.924466 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:54.925772 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:54.926970 9610 hybrid_clock.cc:648] HybridClock initialized: now 1790477094926960 us; error 30 us; skew 500 ppm
I20260927 02:44:54.928896 9610 webserver.cc:460] Webserver started at http://127.0.98.62:37349/ using document root <none> and password file <none>
I20260927 02:44:54.929533 9610 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:54.929625 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:54.929908 9610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:54.932022 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39"
format_stamp: "Formatted at 2026-09-27 02:44:54 on dist-test-slave-gkrl"
server_key: "eddb3384348460467a0e745678b3713f"
server_key_iv: "700ef59210d3275048f0001e386e00ff"
server_key_version: "encryptionkey@0"
I20260927 02:44:54.932591 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39"
format_stamp: "Formatted at 2026-09-27 02:44:54 on dist-test-slave-gkrl"
server_key: "eddb3384348460467a0e745678b3713f"
server_key_iv: "700ef59210d3275048f0001e386e00ff"
server_key_version: "encryptionkey@0"
I20260927 02:44:54.936234 9610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:44:54.938534 9624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:54.940011 9610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:54.940140 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39"
format_stamp: "Formatted at 2026-09-27 02:44:54 on dist-test-slave-gkrl"
server_key: "eddb3384348460467a0e745678b3713f"
server_key_iv: "700ef59210d3275048f0001e386e00ff"
server_key_version: "encryptionkey@0"
I20260927 02:44:54.940258 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:54.952579 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:54.953194 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:54.953382 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:54.959959 9610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43795
I20260927 02:44:54.959983 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43795 every 8 connection(s)
I20260927 02:44:54.960836 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:54.963745 9677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:54.968945 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39: Bootstrap starting.
I20260927 02:44:54.970901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 9610
I20260927 02:44:54.971007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:54.971302 392 external_mini_cluster.cc:1428] Setting key c7f119ae1eae4a6c50245e7c52995b15
I20260927 02:44:54.971455 9677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:54.972296 9677 log.cc:824] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:54.974102 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39: No bootstrap required, opened a new log
I20260927 02:44:54.978221 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43795 } }
I20260927 02:44:54.978425 9677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:54.978459 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8942e3e63e6a4b90a8b407ccb4ec2b39, State: Initialized, Role: FOLLOWER
I20260927 02:44:54.978880 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43795 } }
I20260927 02:44:54.979061 9677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:54.979161 9677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:54.979285 9677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:54.980216 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:54.972498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.0.98.62:43795)
0927 02:44:54.973058 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:54.973073 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:54.973105 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:44:54.973688 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:54.973704 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:54.974002 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:54.974333 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:54.974639 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.975564 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.976807 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:54.977248 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:54.977372 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:54.978229 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:54.978234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:54.978247 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:54.978766 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:54.978822 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:54.978843 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:54.978920 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:54.979114 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 02:44:54.979435 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260927 02:44:54.980237 9677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43795 } }
I20260927 02:44:54.980569 9677 leader_election.cc:302] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8942e3e63e6a4b90a8b407ccb4ec2b39; no voters:
I20260927 02:44:54.980847 9677 leader_election.cc:288] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:54.980948 9682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:54.981145 9682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [term 1 LEADER]: Becoming Leader. State: Replica: 8942e3e63e6a4b90a8b407ccb4ec2b39, State: Running, Role: LEADER
I20260927 02:44:54.982000 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:54.982312 9682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43795 } }
I20260927 02:44:54.984045 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8942e3e63e6a4b90a8b407ccb4ec2b39. Latest consensus state: current_term: 1 leader_uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43795 } } }
I20260927 02:44:54.984162 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:54.984412 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8942e3e63e6a4b90a8b407ccb4ec2b39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43795 } } }
I20260927 02:44:54.984531 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39 [sys.catalog]: This master's current role is: LEADER
W20260927 02:44:54.986629 9696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:44:54.986701 9696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:44:54.986799 9697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:54.987741 9697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:54.992908 9697 catalog_manager.cc:1347] Generated new cluster ID: 88ec7103c1de4300b864464c9096ef56
I20260927 02:44:54.992975 9697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:55.019135 9697 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:55.020125 9697 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:55.029503 9697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8942e3e63e6a4b90a8b407ccb4ec2b39: Generated new TSK 0
I20260927 02:44:55.030177 9697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:55.038743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43795
--builtin_ntp_servers=127.0.98.20:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 02:44:55.157788 9701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:55.158010 9701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:55.158043 9701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:55.161830 9701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:55.161870 9701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:55.161918 9701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:55.166384 9701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:55.167217 9701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:55.168510 9701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:55.177546 9706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:55.177552 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:55.177541 9709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:55.177896 9701 server_base.cc:1034] running on GCE node
I20260927 02:44:55.178260 9701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:55.178843 9701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:55.180009 9701 hybrid_clock.cc:648] HybridClock initialized: now 1790477095180000 us; error 28 us; skew 500 ppm
I20260927 02:44:55.181704 9701 webserver.cc:460] Webserver started at http://127.0.98.1:35819/ using document root <none> and password file <none>
I20260927 02:44:55.182245 9701 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:55.182335 9701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:55.182606 9701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:55.184729 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "3e5b1891ace540e1bd3afeab5ff1881d"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "3383e081161d2e4797e7953d0f265ca4"
server_key_iv: "bd97f5c0fef8a86159134445bac0475d"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.185259 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3e5b1891ace540e1bd3afeab5ff1881d"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "3383e081161d2e4797e7953d0f265ca4"
server_key_iv: "bd97f5c0fef8a86159134445bac0475d"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.188622 9701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:55.190889 9715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.192199 9701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:55.192329 9701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "3e5b1891ace540e1bd3afeab5ff1881d"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "3383e081161d2e4797e7953d0f265ca4"
server_key_iv: "bd97f5c0fef8a86159134445bac0475d"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.192433 9701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:55.218679 9701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:55.219384 9701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:55.219583 9701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:55.220203 9701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:55.221144 9701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:55.221189 9701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.221227 9701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:55.221280 9701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.229768 9701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32931
I20260927 02:44:55.229792 9828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32931 every 8 connection(s)
I20260927 02:44:55.230631 9701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:55.235476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 9701
I20260927 02:44:55.235603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:55.235948 392 external_mini_cluster.cc:1428] Setting key 19a9caab3c37046dbdcdbf17250c768e
I20260927 02:44:55.238045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43795
--builtin_ntp_servers=127.0.98.20:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:44:55.240541 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.232481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55105 (local address 127.0.98.62:43795)
0927 02:44:55.232641 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:55.232646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:55.233668 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 02:44:55.235025 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:55.235027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:55.235071 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:55.235143 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:55.236225 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.236797 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.238137 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.238549 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.238671 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:55.239943 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:55.239950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:55.239954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:55.239971 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:55.239998 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:55.240009 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:55.240086 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:55.240283 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:44:55.240348 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 02:44:55.241242 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.232865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43795 (local address 127.0.98.1:55105)
0927 02:44:55.233529 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:55.233568 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:55.234848 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:55.235304 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:55.235310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:55.235389 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:55.236077 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:55.236095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.236927 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.236931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:55.238002 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:55.238012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.238707 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.238710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:55.238877 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:55.239585 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:55.239587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:55.239637 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:55.239710 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:55.240101 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:55.240113 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:55.240122 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 02:44:55.240239 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:44:55.240451 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
I20260927 02:44:55.242446 9829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43795
I20260927 02:44:55.242694 9829 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:55.243368 9829 heartbeater.cc:511] Master 127.0.98.62:43795 requested a full tablet report, sending...
I20260927 02:44:55.244935 9641 ts_manager.cc:194] Registered new tserver with Master: 3e5b1891ace540e1bd3afeab5ff1881d (127.0.98.1:32931)
I20260927 02:44:55.246057 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55105
W20260927 02:44:55.349591 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:55.349864 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:55.349941 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:55.353940 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:55.354009 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:55.354089 9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:55.358657 9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:55.359611 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:55.361491 9832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:55.370675 9837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:55.370786 9832 server_base.cc:1034] running on GCE node
W20260927 02:44:55.370674 9838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:55.370749 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:55.371419 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:55.371989 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:55.373148 9832 hybrid_clock.cc:648] HybridClock initialized: now 1790477095373138 us; error 23 us; skew 500 ppm
I20260927 02:44:55.374806 9832 webserver.cc:460] Webserver started at http://127.0.98.2:43229/ using document root <none> and password file <none>
I20260927 02:44:55.375348 9832 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:55.375406 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:55.375625 9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:55.377563 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "76e3bae72fad46ae884c7bd3d2ef5f55"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "a0e26edf067b68cff41e4a077412ca5d"
server_key_iv: "742bfb2d76b2162fe531407bb3e1878f"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.378096 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76e3bae72fad46ae884c7bd3d2ef5f55"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "a0e26edf067b68cff41e4a077412ca5d"
server_key_iv: "742bfb2d76b2162fe531407bb3e1878f"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.381578 9832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:55.384106 9846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.385244 9832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:55.385365 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "76e3bae72fad46ae884c7bd3d2ef5f55"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "a0e26edf067b68cff41e4a077412ca5d"
server_key_iv: "742bfb2d76b2162fe531407bb3e1878f"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.385459 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:55.397889 9832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:55.398486 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:55.398670 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:55.399195 9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:55.400116 9832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:55.400164 9832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.400203 9832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:55.400254 9832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.408716 9832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43049
I20260927 02:44:55.408742 9959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43049 every 8 connection(s)
I20260927 02:44:55.409554 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:55.414171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 9832
I20260927 02:44:55.414258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:55.414527 392 external_mini_cluster.cc:1428] Setting key 8ac844f52c5142e5de34602d5e38e077
I20260927 02:44:55.416431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43795
--builtin_ntp_servers=127.0.98.20:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:44:55.418637 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.411221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38237 (local address 127.0.98.62:43795)
0927 02:44:55.411347 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:55.411351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:55.412366 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 02:44:55.413652 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:55.413655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:55.413701 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:55.413774 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:55.414751 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.415392 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.416754 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.416977 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.417085 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:55.418158 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:55.418161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:55.418164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:55.418178 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:55.418202 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:55.418210 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:55.418280 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:55.418440 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 02:44:55.418497 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:44:55.419427 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.411556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43795 (local address 127.0.98.2:38237)
0927 02:44:55.412225 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:55.412263 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:55.413464 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:55.413960 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:55.413967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:55.414046 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:55.414591 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:55.414611 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.415516 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.415519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:55.416629 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:55.416640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.417119 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.417123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:55.417289 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:55.417898 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:55.417900 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:55.417937 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:55.417999 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:55.418293 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:55.418303 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:55.418308 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:44:55.418440 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 02:44:55.418702 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260927 02:44:55.420692 9960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43795
I20260927 02:44:55.420992 9960 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:55.421512 9960 heartbeater.cc:511] Master 127.0.98.62:43795 requested a full tablet report, sending...
I20260927 02:44:55.422557 9641 ts_manager.cc:194] Registered new tserver with Master: 76e3bae72fad46ae884c7bd3d2ef5f55 (127.0.98.2:43049)
I20260927 02:44:55.423159 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38237
W20260927 02:44:55.522993 9963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:55.523263 9963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:55.523344 9963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:55.527334 9963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:55.527431 9963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:55.527520 9963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:55.532326 9963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:55.533304 9963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:55.534590 9963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:55.543512 9968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:55.543527 9969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:55.543522 9971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:55.543725 9963 server_base.cc:1034] running on GCE node
I20260927 02:44:55.544245 9963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:55.544843 9963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:55.546025 9963 hybrid_clock.cc:648] HybridClock initialized: now 1790477095545996 us; error 43 us; skew 500 ppm
I20260927 02:44:55.548091 9963 webserver.cc:460] Webserver started at http://127.0.98.3:44461/ using document root <none> and password file <none>
I20260927 02:44:55.548619 9963 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:55.548678 9963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:55.548895 9963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:55.550771 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "7812d0ebf03c42f08c477a17a6ed562b"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "5f87d3e1d74e4504c6c980305e07dc0a"
server_key_iv: "39c817fe385a34b9edfef32a116d6f1f"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.551311 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7812d0ebf03c42f08c477a17a6ed562b"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "5f87d3e1d74e4504c6c980305e07dc0a"
server_key_iv: "39c817fe385a34b9edfef32a116d6f1f"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.554737 9963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 02:44:55.557174 9977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.558317 9963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:55.558455 9963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "7812d0ebf03c42f08c477a17a6ed562b"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "5f87d3e1d74e4504c6c980305e07dc0a"
server_key_iv: "39c817fe385a34b9edfef32a116d6f1f"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.558561 9963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:55.574003 9963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:55.574607 9963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:55.574790 9963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:55.575353 9963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:55.576279 9963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:55.576354 9963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.576422 9963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:55.576468 9963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.585078 9963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34385
I20260927 02:44:55.585099 10090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34385 every 8 connection(s)
I20260927 02:44:55.585915 9963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:55.592190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 9963
I20260927 02:44:55.592348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:55.592644 392 external_mini_cluster.cc:1428] Setting key 75adf9cbfd646f2eece3aa1a742df620
I20260927 02:44:55.595582 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.587718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48845 (local address 127.0.98.62:43795)
0927 02:44:55.587943 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:55.587947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:55.589078 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 02:44:55.590463 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:55.590467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:55.590524 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:55.590626 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:55.591488 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.592146 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.593564 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.593812 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.593938 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:55.595093 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:55.595097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:55.595100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:55.595118 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:55.595136 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:55.595143 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:55.595221 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:55.595381 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 02:44:55.595442 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:44:55.596551 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.588258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43795 (local address 127.0.98.3:48845)
0927 02:44:55.588904 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:55.588943 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:55.590245 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:55.590788 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:55.590794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:55.590878 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:55.591338 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:55.591349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.592338 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.592342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:55.593428 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:55.593437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:55.593993 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:55.593997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:55.594170 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:55.594839 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:55.594841 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:55.594880 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:55.594942 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:55.595234 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:55.595244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:55.595249 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:44:55.595380 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 02:44:55.595647 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
I20260927 02:44:55.597635 10091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43795
I20260927 02:44:55.597913 10091 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:55.598424 10091 heartbeater.cc:511] Master 127.0.98.62:43795 requested a full tablet report, sending...
I20260927 02:44:55.599376 9641 ts_manager.cc:194] Registered new tserver with Master: 7812d0ebf03c42f08c477a17a6ed562b (127.0.98.3:34385)
I20260927 02:44:55.600013 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48845
I20260927 02:44:55.606411 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:44:55.608919 9681 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:55.608227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.0.98.62:43795)
0927 02:44:55.608344 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:55.608347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:55.608468 (+ 121us) server_negotiation.cc:407] Connection header received
0927 02:44:55.608549 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:55.608552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:55.608586 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:55.608655 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:55.608808 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43592: BlockingRecv error: recv got EOF from 127.0.0.1:43592 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:44:55.609129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 9701
I20260927 02:44:55.614519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 9832
I20260927 02:44:55.619776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 9963
I20260927 02:44:55.624918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 9610
2026-09-27T02:44:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (858 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T02:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:55Z Disabled control of system clock
I20260927 02:44:55.652984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35251
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:44:55.758545 10102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:55.758781 10102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:55.758823 10102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:55.762710 10102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:55.762758 10102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:55.762774 10102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:55.762790 10102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:55.762804 10102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:55.767714 10102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:55.768744 10102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:55.770059 10102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:55.778167 10107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:55.778220 10110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:55.778167 10108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:55.778434 10102 server_base.cc:1034] running on GCE node
I20260927 02:44:55.779023 10102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:55.780189 10102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:55.781358 10102 hybrid_clock.cc:648] HybridClock initialized: now 1790477095781346 us; error 26 us; skew 500 ppm
I20260927 02:44:55.783180 10102 webserver.cc:460] Webserver started at http://127.0.98.62:45753/ using document root <none> and password file <none>
I20260927 02:44:55.783735 10102 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:55.783834 10102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:55.784077 10102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:55.786027 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "209f5de7d00f4df58fea2cdbf4514e6c"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "74a072ad864a366725f063c262c42348"
server_key_iv: "2b6337a24fc8eb875158e827be4df154"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.786553 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "209f5de7d00f4df58fea2cdbf4514e6c"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "74a072ad864a366725f063c262c42348"
server_key_iv: "2b6337a24fc8eb875158e827be4df154"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.790045 10102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:44:55.792338 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:55.793430 10102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:55.793569 10102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "209f5de7d00f4df58fea2cdbf4514e6c"
format_stamp: "Formatted at 2026-09-27 02:44:55 on dist-test-slave-gkrl"
server_key: "74a072ad864a366725f063c262c42348"
server_key_iv: "2b6337a24fc8eb875158e827be4df154"
server_key_version: "encryptionkey@0"
I20260927 02:44:55.793673 10102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:55.798054 10102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:55.798597 10102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:55.798766 10102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:55.805229 10102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35251
I20260927 02:44:55.805238 10168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35251 every 8 connection(s)
I20260927 02:44:55.806082 10102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:55.808120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 10102
I20260927 02:44:55.808238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:55.808539 392 external_mini_cluster.cc:1428] Setting key 5e8a5887ac601c4d0fda49e848ee0962
I20260927 02:44:55.809392 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:55.813812 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.809941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.0.98.62:35251)
0927 02:44:55.810556 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:55.810569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:55.810601 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:44:55.811221 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:55.811236 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:55.811556 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:55.811871 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:55.812662 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:55.812666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:55.812678 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:55.812687 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:55.812739 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:55.812753 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:55.812897 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:55.812968 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:44:55.813187 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260927 02:44:55.815493 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c: Bootstrap starting.
I20260927 02:44:55.817551 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:55.818186 10169 log.cc:824] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:55.820077 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c: No bootstrap required, opened a new log
I20260927 02:44:55.823547 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209f5de7d00f4df58fea2cdbf4514e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35251 } }
I20260927 02:44:55.823751 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:55.823786 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209f5de7d00f4df58fea2cdbf4514e6c, State: Initialized, Role: FOLLOWER
I20260927 02:44:55.824193 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209f5de7d00f4df58fea2cdbf4514e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35251 } }
I20260927 02:44:55.824312 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:55.824357 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:55.824425 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:55.825261 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209f5de7d00f4df58fea2cdbf4514e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35251 } }
I20260927 02:44:55.825544 10169 leader_election.cc:302] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 209f5de7d00f4df58fea2cdbf4514e6c; no voters:
I20260927 02:44:55.825780 10169 leader_election.cc:288] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:55.825907 10174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:55.826149 10174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [term 1 LEADER]: Becoming Leader. State: Replica: 209f5de7d00f4df58fea2cdbf4514e6c, State: Running, Role: LEADER
I20260927 02:44:55.826506 10174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209f5de7d00f4df58fea2cdbf4514e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35251 } }
I20260927 02:44:55.827016 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:55.828159 10175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "209f5de7d00f4df58fea2cdbf4514e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209f5de7d00f4df58fea2cdbf4514e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35251 } } }
I20260927 02:44:55.828293 10175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:55.828684 10183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:55.828644 10176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209f5de7d00f4df58fea2cdbf4514e6c. Latest consensus state: current_term: 1 leader_uuid: "209f5de7d00f4df58fea2cdbf4514e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209f5de7d00f4df58fea2cdbf4514e6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35251 } } }
I20260927 02:44:55.828842 10176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:55.831645 10183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:55.836498 10183 catalog_manager.cc:1347] Generated new cluster ID: f510b44210e140c6806194f1b79cf4d5
I20260927 02:44:55.836556 10183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:55.859840 10183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:55.860169 10183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:55.871727 10183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 209f5de7d00f4df58fea2cdbf4514e6c: Generated new TSK 0
I20260927 02:44:55.872454 10183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:55.881342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35251
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:44:55.988747 10193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:55.989012 10193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:55.989110 10193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:55.993050 10193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:55.993119 10193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:55.993199 10193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:55.997972 10193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:55.998929 10193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:56.000247 10193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:56.011401 10198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:56.011415 10199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:56.011636 10193 server_base.cc:1034] running on GCE node
W20260927 02:44:56.011417 10201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:56.012264 10193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:56.013098 10193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:56.014314 10193 hybrid_clock.cc:648] HybridClock initialized: now 1790477096014300 us; error 27 us; skew 500 ppm
I20260927 02:44:56.016333 10193 webserver.cc:460] Webserver started at http://127.0.98.1:41885/ using document root <none> and password file <none>
I20260927 02:44:56.016923 10193 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:56.017014 10193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:56.017236 10193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:56.019315 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ecdf83bef6643fd98743764428523d2"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "e4710cdbb58f6c942f31edf239a70620"
server_key_iv: "c71160594e69cc3f6d3c02cd5998a4c0"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.019918 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ecdf83bef6643fd98743764428523d2"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "e4710cdbb58f6c942f31edf239a70620"
server_key_iv: "c71160594e69cc3f6d3c02cd5998a4c0"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.023474 10193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:56.025885 10207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.027011 10193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:44:56.027138 10193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "4ecdf83bef6643fd98743764428523d2"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "e4710cdbb58f6c942f31edf239a70620"
server_key_iv: "c71160594e69cc3f6d3c02cd5998a4c0"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.027247 10193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:56.031718 10193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:56.032330 10193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:56.032502 10193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:56.033007 10193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:56.033872 10193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:56.033941 10193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.034008 10193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:56.034063 10193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.042871 10193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44517
I20260927 02:44:56.042900 10320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44517 every 8 connection(s)
I20260927 02:44:56.043685 10193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:56.047411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 10193
I20260927 02:44:56.047537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:56.047833 392 external_mini_cluster.cc:1428] Setting key ce5b26f19fa546be051bc7d8138d2c0a
I20260927 02:44:56.049746 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.045375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54189 (local address 127.0.98.62:35251)
0927 02:44:56.045532 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:56.045536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:56.046557 (+ 1021us) server_negotiation.cc:407] Connection header received
0927 02:44:56.047865 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:56.047868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:56.047945 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:56.048029 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:56.049270 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:56.049274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:56.049277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:56.049290 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:56.049312 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:56.049322 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:56.049413 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:56.049542 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 02:44:56.049583 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:44:56.049923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35251
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 02:44:56.050594 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.045797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35251 (local address 127.0.98.1:54189)
0927 02:44:56.046418 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:56.046455 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:56.047653 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:56.048210 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:56.048219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:56.048334 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:56.048958 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:56.048965 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:56.049024 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:56.049112 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:56.049410 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:56.049417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:56.049421 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:56.049549 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 02:44:56.049722 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260927 02:44:56.051700 10321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35251
I20260927 02:44:56.051937 10321 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:56.052302 10321 heartbeater.cc:511] Master 127.0.98.62:35251 requested a full tablet report, sending...
I20260927 02:44:56.053691 10133 ts_manager.cc:194] Registered new tserver with Master: 4ecdf83bef6643fd98743764428523d2 (127.0.98.1:44517)
W20260927 02:44:56.156533 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:56.156792 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:56.156872 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:56.160838 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:56.160905 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:56.160992 10324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:56.165658 10324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:56.166623 10324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:56.168010 10324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:56.177356 10332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:56.177335 10329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:56.177393 10324 server_base.cc:1034] running on GCE node
W20260927 02:44:56.177358 10330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:56.177932 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:56.178494 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:56.179653 10324 hybrid_clock.cc:648] HybridClock initialized: now 1790477096179620 us; error 46 us; skew 500 ppm
I20260927 02:44:56.181476 10324 webserver.cc:460] Webserver started at http://127.0.98.2:46743/ using document root <none> and password file <none>
I20260927 02:44:56.182057 10324 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:56.182147 10324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:56.182358 10324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:56.184410 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "b637474ce70e4befaa5848e15e164b1f"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "ab2804c3f0bbda78107f9cfb87e5e71b"
server_key_iv: "b917664446281fd3e1ec9bc26e290dea"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.184948 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b637474ce70e4befaa5848e15e164b1f"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "ab2804c3f0bbda78107f9cfb87e5e71b"
server_key_iv: "b917664446281fd3e1ec9bc26e290dea"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.188464 10324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:44:56.190869 10338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.192101 10324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:56.192224 10324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "b637474ce70e4befaa5848e15e164b1f"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "ab2804c3f0bbda78107f9cfb87e5e71b"
server_key_iv: "b917664446281fd3e1ec9bc26e290dea"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.192333 10324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:56.197271 10324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:56.197999 10324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:56.198172 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:56.198731 10324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:56.199715 10324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:56.199757 10324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.199812 10324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:56.199832 10324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.209297 10324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45251
I20260927 02:44:56.209393 10451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45251 every 8 connection(s)
I20260927 02:44:56.210250 10324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:56.216051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 10324
I20260927 02:44:56.216152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:56.216482 392 external_mini_cluster.cc:1428] Setting key 81022ee9da91f0523a55b6d1adcfcd31
I20260927 02:44:56.216976 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.212257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35047 (local address 127.0.98.62:35251)
0927 02:44:56.212401 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:56.212405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:56.213533 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 02:44:56.214870 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:56.214873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:56.214918 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:56.215023 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:56.216436 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:56.216440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:56.216443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:56.216462 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:56.216481 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:56.216489 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:56.216601 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:56.216787 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 02:44:56.216827 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:44:56.217945 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.212684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35251 (local address 127.0.98.2:35047)
0927 02:44:56.213369 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:56.213409 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:56.214623 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:56.215190 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:56.215196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:56.215288 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:56.216095 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:56.216097 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:56.216169 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:56.216287 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:56.216611 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:56.216622 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:56.216628 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:44:56.216827 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 02:44:56.217106 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260927 02:44:56.218502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35251
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 02:44:56.219175 10452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35251
I20260927 02:44:56.219439 10452 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:56.220005 10452 heartbeater.cc:511] Master 127.0.98.62:35251 requested a full tablet report, sending...
I20260927 02:44:56.221164 10133 ts_manager.cc:194] Registered new tserver with Master: b637474ce70e4befaa5848e15e164b1f (127.0.98.2:45251)
W20260927 02:44:56.326540 10455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:56.326822 10455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:56.326908 10455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:56.331292 10455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:56.331360 10455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:56.331461 10455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:56.336154 10455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:56.337064 10455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:56.338295 10455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:56.347520 10461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:56.347592 10463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:56.347620 10460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:56.347822 10455 server_base.cc:1034] running on GCE node
I20260927 02:44:56.348248 10455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:56.348906 10455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:56.350087 10455 hybrid_clock.cc:648] HybridClock initialized: now 1790477096350060 us; error 49 us; skew 500 ppm
I20260927 02:44:56.351759 10455 webserver.cc:460] Webserver started at http://127.0.98.3:37987/ using document root <none> and password file <none>
I20260927 02:44:56.352361 10455 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:56.352450 10455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:56.352679 10455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:56.354593 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "8e9e3a66ad524d35826104acd45c7a1f"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "2e2d17b71ae4ca6f98e791e20c15f011"
server_key_iv: "dec6de8600e5db44438d160b1c5ef207"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.355130 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8e9e3a66ad524d35826104acd45c7a1f"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "2e2d17b71ae4ca6f98e791e20c15f011"
server_key_iv: "dec6de8600e5db44438d160b1c5ef207"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.358510 10455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 02:44:56.360906 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.362068 10455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:56.362200 10455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "8e9e3a66ad524d35826104acd45c7a1f"
format_stamp: "Formatted at 2026-09-27 02:44:56 on dist-test-slave-gkrl"
server_key: "2e2d17b71ae4ca6f98e791e20c15f011"
server_key_iv: "dec6de8600e5db44438d160b1c5ef207"
server_key_version: "encryptionkey@0"
I20260927 02:44:56.362308 10455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:56.366658 10455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:56.367221 10455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:56.367413 10455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:56.367988 10455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:56.368809 10455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:56.368880 10455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.368947 10455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:56.368981 10455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:56.377959 10455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33713
I20260927 02:44:56.378010 10582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33713 every 8 connection(s)
I20260927 02:44:56.378849 10455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:56.384567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 10455
I20260927 02:44:56.384697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:56.385106 392 external_mini_cluster.cc:1428] Setting key 04073d9d30cee045b2cdbbc8263fda3b
I20260927 02:44:56.385285 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.380836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58713 (local address 127.0.98.62:35251)
0927 02:44:56.380977 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:56.380982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:56.382079 (+ 1097us) server_negotiation.cc:407] Connection header received
0927 02:44:56.383482 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:56.383485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:56.383520 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:56.383594 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:56.384769 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:56.384774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:56.384776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:44:56.384791 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:56.384818 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:44:56.384827 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:56.384910 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:56.385088 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 02:44:56.385124 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:44:56.386234 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.381277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35251 (local address 127.0.98.3:58713)
0927 02:44:56.381936 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:56.381977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:56.383289 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:56.383722 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:56.383728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:56.383809 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:56.384428 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:44:56.384430 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:56.384484 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:44:56.384573 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:56.384922 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:56.384931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:56.384938 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:44:56.385115 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 02:44:56.385359 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
I20260927 02:44:56.387298 10583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35251
I20260927 02:44:56.387532 10583 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:56.387960 10583 heartbeater.cc:511] Master 127.0.98.62:35251 requested a full tablet report, sending...
I20260927 02:44:56.388983 10132 ts_manager.cc:194] Registered new tserver with Master: 8e9e3a66ad524d35826104acd45c7a1f (127.0.98.3:33713)
I20260927 02:44:56.398815 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:44:56.401799 10172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:56.400836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.0.98.62:35251)
0927 02:44:56.401003 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:56.401009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:56.401144 (+ 135us) server_negotiation.cc:407] Connection header received
0927 02:44:56.401305 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:56.401308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:56.401355 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:56.401441 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:56.401695 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59256: BlockingRecv error: recv got EOF from 127.0.0.1:59256 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 02:44:56.402168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 10193
I20260927 02:44:56.407824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 10324
I20260927 02:44:56.412781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 10455
I20260927 02:44:56.417692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 10102
2026-09-27T02:44:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (792 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:56 dist-test-slave-gkrl krb5kdc[10593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:56 dist-test-slave-gkrl krb5kdc[10593](info): set up 2 sockets
Sep 27 02:44:56 dist-test-slave-gkrl krb5kdc[10593](info): commencing operation
krb5kdc: starting...
W20260927 02:44:57.476831 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:57 dist-test-slave-gkrl krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:57.595423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42303
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:57.699331 10606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:57.699563 10606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:57.699600 10606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:57.703673 10606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:57.703727 10606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:57.703745 10606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:57.703761 10606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:57.703819 10606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:57.709002 10606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:57.710003 10606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:57.710981 10606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:57.719201 10611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:57.719262 10612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:57.719208 10614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:57.719378 10606 server_base.cc:1034] running on GCE node
I20260927 02:44:57.720091 10606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:57.721279 10606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:57.722445 10606 hybrid_clock.cc:648] HybridClock initialized: now 1790477097722422 us; error 42 us; skew 500 ppm
Sep 27 02:44:57 dist-test-slave-gkrl krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:57.727726 10606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:57.728626 10606 webserver.cc:460] Webserver started at http://127.0.98.62:39155/ using document root <none> and password file <none>
I20260927 02:44:57.729167 10606 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:57.729269 10606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:57.729508 10606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:57.731438 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9"
format_stamp: "Formatted at 2026-09-27 02:44:57 on dist-test-slave-gkrl"
server_key: "aa5ca3ed44a54f84746b4885e7d87bc6"
server_key_iv: "e32399628bad120dacf2ad0bf0798230"
server_key_version: "encryptionkey@0"
I20260927 02:44:57.732074 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9"
format_stamp: "Formatted at 2026-09-27 02:44:57 on dist-test-slave-gkrl"
server_key: "aa5ca3ed44a54f84746b4885e7d87bc6"
server_key_iv: "e32399628bad120dacf2ad0bf0798230"
server_key_version: "encryptionkey@0"
I20260927 02:44:57.735502 10606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:44:57.737917 10621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:57.739144 10606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:44:57.739305 10606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9"
format_stamp: "Formatted at 2026-09-27 02:44:57 on dist-test-slave-gkrl"
server_key: "aa5ca3ed44a54f84746b4885e7d87bc6"
server_key_iv: "e32399628bad120dacf2ad0bf0798230"
server_key_version: "encryptionkey@0"
I20260927 02:44:57.739418 10606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:57.758688 10606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:57.762524 10606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:57.762761 10606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:57.769562 10606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42303
I20260927 02:44:57.769845 10673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42303 every 8 connection(s)
I20260927 02:44:57.770567 10606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:57.770994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 10606
I20260927 02:44:57.771239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:57.771544 392 external_mini_cluster.cc:1428] Setting key 807689c76e8f65ae5e4162afcdf251ec
I20260927 02:44:57.774024 10674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:57.779341 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9: Bootstrap starting.
Sep 27 02:44:57 dist-test-slave-gkrl krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:57.781734 10674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:57.782428 10674 log.cc:824] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:57.784379 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9: No bootstrap required, opened a new log
I20260927 02:44:57.785096 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:57.772976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.0.98.62:42303)
0927 02:44:57.773411 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:57.773426 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:57.773462 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:44:57.774132 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:57.774155 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:57.774486 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:57.774820 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:57.775627 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:57.776662 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:57.777504 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:57.777960 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:57.778133 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:57.780643 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:57.780647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:57.780659 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:57.780684 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:57.782432 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:57.782856 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:57.782863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:57.782874 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:57.782977 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:57.783257 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:57.783261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:57.783264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:57.783628 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:57.783826 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:57.784100 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 02:44:57.784373 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":136,"threads_started":1}
I20260927 02:44:57.788069 10674 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42303 } }
I20260927 02:44:57.788340 10674 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:57.788429 10674 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1cd67bf4ca4bd6a2456403f3a7e6f9, State: Initialized, Role: FOLLOWER
I20260927 02:44:57.788829 10674 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42303 } }
I20260927 02:44:57.788975 10674 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:57.789062 10674 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:57.789179 10674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:57.790033 10674 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42303 } }
I20260927 02:44:57.790386 10674 leader_election.cc:302] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf1cd67bf4ca4bd6a2456403f3a7e6f9; no voters:
I20260927 02:44:57.790680 10674 leader_election.cc:288] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:57.790808 10679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:57.791061 10679 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [term 1 LEADER]: Becoming Leader. State: Replica: bf1cd67bf4ca4bd6a2456403f3a7e6f9, State: Running, Role: LEADER
I20260927 02:44:57.791364 10679 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42303 } }
I20260927 02:44:57.791985 10674 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:57.793088 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf1cd67bf4ca4bd6a2456403f3a7e6f9. Latest consensus state: current_term: 1 leader_uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42303 } } }
I20260927 02:44:57.793093 10680 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1cd67bf4ca4bd6a2456403f3a7e6f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42303 } } }
I20260927 02:44:57.793244 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:57.793287 10680 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:57.793672 10688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:57.796674 10688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:57.802820 10688 catalog_manager.cc:1347] Generated new cluster ID: 05cc598f0f6049eaba55d78c0d7e60ee
I20260927 02:44:57.802898 10688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:57.821705 10688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:57.823076 10688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:57.840559 10688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9: Generated new TSK 0
I20260927 02:44:57.841429 10688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:57.879369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42303
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:57.986737 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:57.986981 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:57.987057 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:57.991063 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:57.991114 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:57.991225 10700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:57.996570 10700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:57.997535 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:57.998513 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:58.009152 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:58.009173 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:58.009541 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:58.009624 10700 server_base.cc:1034] running on GCE node
I20260927 02:44:58.010128 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:58.010914 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:58.012091 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790477098012064 us; error 49 us; skew 500 ppm
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:58.014969 10700 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:58.015741 10700 webserver.cc:460] Webserver started at http://127.0.98.1:43287/ using document root <none> and password file <none>
I20260927 02:44:58.016320 10700 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:58.016383 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:58.016541 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:58.018443 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "5bbc7a0596974890a26f00d8e1fee9bc"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "bd069d227ea09448125463001789296c"
server_key_iv: "071afd08ecd6a40253261aeab08fc536"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.019002 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5bbc7a0596974890a26f00d8e1fee9bc"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "bd069d227ea09448125463001789296c"
server_key_iv: "071afd08ecd6a40253261aeab08fc536"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.022519 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:44:58.024886 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.026036 10700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:58.026129 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "5bbc7a0596974890a26f00d8e1fee9bc"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "bd069d227ea09448125463001789296c"
server_key_iv: "071afd08ecd6a40253261aeab08fc536"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.026206 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:58.038688 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:58.041431 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:58.041568 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:58.042047 10700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:58.042866 10700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:58.042910 10700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.042948 10700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:58.042963 10700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.051478 10700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36035
I20260927 02:44:58.051499 10828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36035 every 8 connection(s)
I20260927 02:44:58.052356 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:58.055325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 10700
I20260927 02:44:58.055462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:58.055750 392 external_mini_cluster.cc:1428] Setting key 972cb708548abe62387e492a3da30346
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:44:58.065469 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.054071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38129 (local address 127.0.98.62:42303)
0927 02:44:58.054200 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.054203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.055176 (+ 973us) server_negotiation.cc:407] Connection header received
0927 02:44:58.056276 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.056279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.056336 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.056451 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:58.057546 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.058066 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.059177 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.059411 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.059527 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.061907 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:58.061914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:58.061918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:58.061939 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:58.063385 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:58.063915 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:58.063922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:58.063924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:58.064004 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:58.064340 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:58.064345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:58.064347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:58.064561 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:58.064663 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.065139 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.065263 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:44:58.066248 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.054436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42303 (local address 127.0.98.1:38129)
0927 02:44:58.055035 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.055074 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:58.056076 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:58.056619 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:58.056625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:58.056956 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:58.057407 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.057420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.058192 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.058196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.059051 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.059060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.059559 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.059563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.059723 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.060338 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:58.060372 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:58.061742 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:58.063502 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:58.063508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:58.063523 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:58.063765 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:58.064103 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:58.064108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:58.064110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:58.064244 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:58.064674 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:58.064681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:58.064911 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 02:44:58.065098 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 02:44:58.065407 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:44:58.067556 10829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42303
I20260927 02:44:58.067806 10829 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:58.068377 10829 heartbeater.cc:511] Master 127.0.98.62:42303 requested a full tablet report, sending...
I20260927 02:44:58.070043 10638 ts_manager.cc:194] Registered new tserver with Master: 5bbc7a0596974890a26f00d8e1fee9bc (127.0.98.1:36035)
I20260927 02:44:58.071697 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:58.083457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42303
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:58.190505 10834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:58.190737 10834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:58.190778 10834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:58.194854 10834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:58.194904 10834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:58.194985 10834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:58.199927 10834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:58.200873 10834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:58.201834 10834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:58.211762 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:58.211783 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:58.211946 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:58.212845 10834 server_base.cc:1034] running on GCE node
I20260927 02:44:58.213289 10834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:58.213948 10834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:58.215159 10834 hybrid_clock.cc:648] HybridClock initialized: now 1790477098215127 us; error 42 us; skew 500 ppm
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:58.218127 10834 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:58.218972 10834 webserver.cc:460] Webserver started at http://127.0.98.2:44085/ using document root <none> and password file <none>
I20260927 02:44:58.219501 10834 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:58.219561 10834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:58.219724 10834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:58.221871 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "957628ef1440486b9716014880d56b61"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "327dc1da36865d25f1167c5a71221bf5"
server_key_iv: "ea3b4e1802d12a7f9c07fe003b12a96d"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.222399 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "957628ef1440486b9716014880d56b61"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "327dc1da36865d25f1167c5a71221bf5"
server_key_iv: "ea3b4e1802d12a7f9c07fe003b12a96d"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.226254 10834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 02:44:58.228817 10849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.230130 10834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:44:58.230285 10834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "957628ef1440486b9716014880d56b61"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "327dc1da36865d25f1167c5a71221bf5"
server_key_iv: "ea3b4e1802d12a7f9c07fe003b12a96d"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.230386 10834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:58.268728 10834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:58.271564 10834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:58.271708 10834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:58.272349 10834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:58.273093 10834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:58.273138 10834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.273182 10834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:58.273204 10834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.281761 10834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46331
I20260927 02:44:58.281780 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46331 every 8 connection(s)
I20260927 02:44:58.282627 10834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:58.289837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 10834
I20260927 02:44:58.289984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:58.290305 392 external_mini_cluster.cc:1428] Setting key 1857ebf01cac770fdb3c56705b0831df
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:58.296536 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.284407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33809 (local address 127.0.98.62:42303)
0927 02:44:58.284544 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.284547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.285623 (+ 1076us) server_negotiation.cc:407] Connection header received
0927 02:44:58.286432 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.286435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.286481 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.286552 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:58.287753 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.288338 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.289485 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.289695 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.289815 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.292435 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:58.292439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:58.292441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:58.292461 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:58.294321 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:58.294839 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:58.294846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:58.294850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:58.294943 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:58.295267 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:58.295273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:58.295278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:58.295515 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:58.295623 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.296207 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.296349 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:44:58.297326 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.284852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42303 (local address 127.0.98.2:33809)
0927 02:44:58.285479 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.285520 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:58.286243 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:58.286709 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:58.286715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:58.287143 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:58.287621 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.287633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.288473 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.288476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.289368 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.289376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.289850 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.289854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.290069 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.290648 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:58.290663 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:58.292280 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:58.294447 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:58.294452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:58.294463 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:58.294724 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:58.295037 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:58.295039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:58.295040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:58.295166 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:58.295644 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:58.295650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:58.295929 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 02:44:58.296124 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 02:44:58.296379 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":103,"threads_started":1}
I20260927 02:44:58.298491 10963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42303
I20260927 02:44:58.298738 10963 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:58.299324 10963 heartbeater.cc:511] Master 127.0.98.62:42303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:44:58.300518 10638 ts_manager.cc:194] Registered new tserver with Master: 957628ef1440486b9716014880d56b61 (127.0.98.2:46331)
I20260927 02:44:58.301090 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:58.319290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42303
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:58.428148 10968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:58.428418 10968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:58.428496 10968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:58.432622 10968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:58.432710 10968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:58.432807 10968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:58.439010 10968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:44:58.440079 10968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:58.441052 10968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:58.450333 10974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:58.450320 10973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:58.450325 10976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:58.450440 10968 server_base.cc:1034] running on GCE node
I20260927 02:44:58.450910 10968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:58.451539 10968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:58.452694 10968 hybrid_clock.cc:648] HybridClock initialized: now 1790477098452662 us; error 42 us; skew 500 ppm
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:58.455353 10968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:58.456262 10968 webserver.cc:460] Webserver started at http://127.0.98.3:40463/ using document root <none> and password file <none>
I20260927 02:44:58.456828 10968 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:58.456925 10968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:58.457145 10968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:58.459030 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "58d7cce947e44d2f95ba57d49e10a40a"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "f7915817aa9d141b9af660c2630986b9"
server_key_iv: "321faebb03bf66a7c3a82c5eb6cb5ad0"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.459573 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "58d7cce947e44d2f95ba57d49e10a40a"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "f7915817aa9d141b9af660c2630986b9"
server_key_iv: "321faebb03bf66a7c3a82c5eb6cb5ad0"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.463073 10968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:44:58.465372 10983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.466655 10968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:58.466807 10968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "58d7cce947e44d2f95ba57d49e10a40a"
format_stamp: "Formatted at 2026-09-27 02:44:58 on dist-test-slave-gkrl"
server_key: "f7915817aa9d141b9af660c2630986b9"
server_key_iv: "321faebb03bf66a7c3a82c5eb6cb5ad0"
server_key_version: "encryptionkey@0"
I20260927 02:44:58.466965 10968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:58.488421 10968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:58.491912 10968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:58.492125 10968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:58.492815 10968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:58.493665 10968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:58.493716 10968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.493789 10968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:58.493829 10968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:58.502945 10968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35015
I20260927 02:44:58.503291 11096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35015 every 8 connection(s)
I20260927 02:44:58.503906 10968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:58.505209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 10968
I20260927 02:44:58.505342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:58.505641 392 external_mini_cluster.cc:1428] Setting key ddbb723d80b73e31b0dc4ae84923ac93
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:58.517200 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.505846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58837 (local address 127.0.98.62:42303)
0927 02:44:58.506017 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.506020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.507066 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 02:44:58.508130 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.508133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.508177 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.508246 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:58.509311 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.509833 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.510932 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.511140 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.511222 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.513646 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:58.513650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:58.513652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:58.513675 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:58.515105 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:58.515639 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:58.515644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:58.515646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:58.515728 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:58.516124 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:58.516128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:58.516129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:58.516310 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:58.516429 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.516915 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.517031 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:44:58.517982 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.506366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42303 (local address 127.0.98.3:58837)
0927 02:44:58.506903 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.506946 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:58.507948 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:58.508380 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:58.508386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:58.508724 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:58.509190 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.509202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.509944 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.509947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.510828 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.510837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.511272 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.511275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.511408 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.512022 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:58.512037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:58.513476 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:58.515261 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:58.515266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:58.515281 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:58.515534 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:58.515830 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:58.515833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:58.515834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:58.516021 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:58.516426 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:58.516432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:58.516675 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 02:44:58.516862 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 02:44:58.517174 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":157,"threads_started":1}
I20260927 02:44:58.519109 11097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42303
I20260927 02:44:58.519344 11097 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:58.519857 11097 heartbeater.cc:511] Master 127.0.98.62:42303 requested a full tablet report, sending...
I20260927 02:44:58.520864 10637 ts_manager.cc:194] Registered new tserver with Master: 58d7cce947e44d2f95ba57d49e10a40a (127.0.98.3:35015)
I20260927 02:44:58.521436 10637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58837
I20260927 02:44:58.531447 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:58.540771 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.533532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.0.98.62:42303)
0927 02:44:58.533657 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.533662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.533812 (+ 150us) server_negotiation.cc:407] Connection header received
0927 02:44:58.533901 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.533903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.533939 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.534004 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:58.534829 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.535316 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.536232 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.536446 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.536567 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.537875 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:58.537878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:58.537880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:58.537900 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:58.539299 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:58.539687 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:58.539690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:58.539691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:58.539756 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:58.540052 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:58.540054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:58.540056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:58.540240 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:58.540356 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.540501 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.540581 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:44:58.543562 10637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:58.546540 10637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:58.560642 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.554646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36035 (local address 127.0.0.1:55656)
0927 02:44:58.555202 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.555253 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:58.555386 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:58.556446 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:58.556449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:58.556462 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:58.556702 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.556710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.557757 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.557763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.558988 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.558997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.560064 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.560068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.560192 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.560232 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:44:58.560347 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 02:44:58.560461 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":73,"spinlock_wait_cycles":13824,"thread_start_us":144,"threads_started":1}
I20260927 02:44:58.560936 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.555350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.0.98.2:46331)
0927 02:44:58.555718 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.555743 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.555799 (+ 56us) server_negotiation.cc:407] Connection header received
0927 02:44:58.556019 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.556026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.556236 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.556388 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:58.556808 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.558033 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.559377 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.560219 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.560347 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.560474 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.560714 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.560823 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":142,"threads_started":1}
I20260927 02:44:58.561132 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.555106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.98.1:36035)
0927 02:44:58.555914 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.555931 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.555959 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:44:58.556040 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.556047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.556267 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.556591 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:58.556843 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.557614 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.559128 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.559939 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.560069 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.560227 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.560559 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.560653 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":83,"threads_started":1}
I20260927 02:44:58.561214 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.554606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36136 (local address 127.0.98.3:35015)
0927 02:44:58.556153 (+ 1547us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.556164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.556183 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:44:58.556258 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.556263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.556414 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.556546 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:58.556962 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.558067 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.559515 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.560275 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.560364 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.560474 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.560958 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.561069 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1455,"thread_start_us":94,"threads_started":1}
I20260927 02:44:58.561237 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.554901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46331 (local address 127.0.0.1:37740)
0927 02:44:58.555720 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.555735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:58.555827 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:58.556410 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:58.556420 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:58.556448 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:58.556690 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.556699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.558200 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.558204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.559257 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.559263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.560349 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.560354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.560438 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.560461 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:44:58.560550 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:44:58.560609 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":39,"thread_start_us":107,"threads_started":1}
I20260927 02:44:58.561553 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.554447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.0.1:36136)
0927 02:44:58.555202 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.555256 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:58.555406 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:58.556714 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:58.556717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:58.556733 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:58.556837 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.556843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.558231 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.558235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.559386 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.559397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.560537 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.560541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.560626 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.560647 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:44:58.560717 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:44:58.561422 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"spinlock_wait_cycles":7296,"thread_start_us":79,"threads_started":1}
I20260927 02:44:58.564067 11031 tablet_service.cc:1467] Processing CreateTablet for tablet a156627a5523466d9b486c26ec7a1dbe (DEFAULT_TABLE table=test-table [id=53991bd063f949e2876386080616e315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:58.564074 10897 tablet_service.cc:1467] Processing CreateTablet for tablet a156627a5523466d9b486c26ec7a1dbe (DEFAULT_TABLE table=test-table [id=53991bd063f949e2876386080616e315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:58.565009 11031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a156627a5523466d9b486c26ec7a1dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:58.565052 10897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a156627a5523466d9b486c26ec7a1dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:58.565636 10763 tablet_service.cc:1467] Processing CreateTablet for tablet a156627a5523466d9b486c26ec7a1dbe (DEFAULT_TABLE table=test-table [id=53991bd063f949e2876386080616e315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:58.566571 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a156627a5523466d9b486c26ec7a1dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:58.570647 11114 tablet_bootstrap.cc:492] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: Bootstrap starting.
I20260927 02:44:58.572451 11115 tablet_bootstrap.cc:492] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: Bootstrap starting.
I20260927 02:44:58.572925 11114 tablet_bootstrap.cc:654] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:58.573355 11116 tablet_bootstrap.cc:492] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc: Bootstrap starting.
I20260927 02:44:58.574354 11114 log.cc:824] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:58.574929 11115 tablet_bootstrap.cc:654] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:58.575181 11116 tablet_bootstrap.cc:654] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:58.575748 11115 log.cc:824] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:58.575879 11116 log.cc:824] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:58.576409 11114 tablet_bootstrap.cc:492] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: No bootstrap required, opened a new log
I20260927 02:44:58.576596 11114 ts_tablet_manager.cc:1397] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:44:58.577446 11116 tablet_bootstrap.cc:492] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc: No bootstrap required, opened a new log
I20260927 02:44:58.577641 11116 ts_tablet_manager.cc:1397] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 02:44:58.577701 11115 tablet_bootstrap.cc:492] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: No bootstrap required, opened a new log
I20260927 02:44:58.577862 11115 ts_tablet_manager.cc:1397] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:44:58.580283 11114 raft_consensus.cc:348] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.580546 11114 raft_consensus.cc:374] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:58.580601 11114 raft_consensus.cc:729] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d7cce947e44d2f95ba57d49e10a40a, State: Initialized, Role: FOLLOWER
I20260927 02:44:58.581028 11114 consensus_queue.cc:260] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.581104 11116 raft_consensus.cc:348] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.581365 11116 raft_consensus.cc:374] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:58.581427 11116 raft_consensus.cc:729] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbc7a0596974890a26f00d8e1fee9bc, State: Initialized, Role: FOLLOWER
I20260927 02:44:58.581441 11115 raft_consensus.cc:348] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.581661 11115 raft_consensus.cc:374] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:58.581704 11115 raft_consensus.cc:729] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957628ef1440486b9716014880d56b61, State: Initialized, Role: FOLLOWER
I20260927 02:44:58.581868 11116 consensus_queue.cc:260] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.581974 11114 ts_tablet_manager.cc:1428] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:44:58.582089 11115 consensus_queue.cc:260] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.582290 11097 heartbeater.cc:503] Master 127.0.98.62:42303 was elected leader, sending a full tablet report...
I20260927 02:44:58.582690 11116 ts_tablet_manager.cc:1428] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260927 02:44:58.582907 11115 ts_tablet_manager.cc:1428] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:44:58.583022 10829 heartbeater.cc:503] Master 127.0.98.62:42303 was elected leader, sending a full tablet report...
I20260927 02:44:58.583185 10963 heartbeater.cc:503] Master 127.0.98.62:42303 was elected leader, sending a full tablet report...
I20260927 02:44:58.669525 11122 raft_consensus.cc:482] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:58.669838 11122 raft_consensus.cc:504] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.671137 11122 leader_election.cc:288] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58d7cce947e44d2f95ba57d49e10a40a (127.0.98.3:35015), 5bbc7a0596974890a26f00d8e1fee9bc (127.0.98.1:36035)
I20260927 02:44:58.675714 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.671482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43539 (local address 127.0.98.1:36035)
0927 02:44:58.671617 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.671622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.671638 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:44:58.671745 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.671749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.671809 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.671916 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:58.672251 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.673018 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.674536 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.675139 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.675255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.675331 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.675495 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.675566 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:58.675693 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.671397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36035 (local address 127.0.98.2:43539)
0927 02:44:58.671516 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.671534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:58.671625 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:58.671945 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:58.671948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:58.671968 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:58.672104 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.672111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.673137 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.673140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.674414 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.674423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.675256 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.675258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.675342 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.675361 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:44:58.675462 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 02:44:58.675537 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":22}
I20260927 02:44:58.675861 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.671398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.98.2:51443)
0927 02:44:58.672024 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.672038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:58.672134 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:58.672437 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:58.672439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:58.672456 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:58.672544 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.672548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.673445 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.673448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.674608 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:58.674614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.675449 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.675455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:58.675552 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.675571 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:44:58.675642 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:44:58.675696 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":328,"thread_start_us":171,"threads_started":1}
I20260927 02:44:58.675935 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.671466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51443 (local address 127.0.98.3:35015)
0927 02:44:58.671631 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.671635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.672145 (+ 510us) server_negotiation.cc:407] Connection header received
0927 02:44:58.672239 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.672242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.672295 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.672386 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:58.672665 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.673326 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.674726 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.675327 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.675416 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.675497 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.675673 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.675788 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 02:44:58.676352 10783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a156627a5523466d9b486c26ec7a1dbe" candidate_uuid: "957628ef1440486b9716014880d56b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" is_pre_election: true
I20260927 02:44:58.676653 10783 raft_consensus.cc:2454] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957628ef1440486b9716014880d56b61 in term 0.
I20260927 02:44:58.676695 11051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a156627a5523466d9b486c26ec7a1dbe" candidate_uuid: "957628ef1440486b9716014880d56b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d7cce947e44d2f95ba57d49e10a40a" is_pre_election: true
I20260927 02:44:58.676972 11051 raft_consensus.cc:2454] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957628ef1440486b9716014880d56b61 in term 0.
I20260927 02:44:58.677191 10851 leader_election.cc:302] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bbc7a0596974890a26f00d8e1fee9bc, 957628ef1440486b9716014880d56b61; no voters:
I20260927 02:44:58.677474 11122 raft_consensus.cc:2781] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:58.677556 11122 raft_consensus.cc:482] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:58.677597 11122 raft_consensus.cc:3037] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:58.678682 11122 raft_consensus.cc:504] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.679041 11122 leader_election.cc:288] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [CANDIDATE]: Term 1 election: Requested vote from peers 58d7cce947e44d2f95ba57d49e10a40a (127.0.98.3:35015), 5bbc7a0596974890a26f00d8e1fee9bc (127.0.98.1:36035)
I20260927 02:44:58.679338 11051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a156627a5523466d9b486c26ec7a1dbe" candidate_uuid: "957628ef1440486b9716014880d56b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d7cce947e44d2f95ba57d49e10a40a"
I20260927 02:44:58.679461 11051 raft_consensus.cc:3037] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:58.679487 10783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a156627a5523466d9b486c26ec7a1dbe" candidate_uuid: "957628ef1440486b9716014880d56b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbc7a0596974890a26f00d8e1fee9bc"
I20260927 02:44:58.679611 10783 raft_consensus.cc:3037] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:58.680444 11051 raft_consensus.cc:2454] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957628ef1440486b9716014880d56b61 in term 1.
I20260927 02:44:58.680745 10783 raft_consensus.cc:2454] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957628ef1440486b9716014880d56b61 in term 1.
I20260927 02:44:58.680735 10850 leader_election.cc:302] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d7cce947e44d2f95ba57d49e10a40a, 957628ef1440486b9716014880d56b61; no voters:
I20260927 02:44:58.680917 11122 raft_consensus.cc:2781] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:58.681146 11122 raft_consensus.cc:686] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 1 LEADER]: Becoming Leader. State: Replica: 957628ef1440486b9716014880d56b61, State: Running, Role: LEADER
I20260927 02:44:58.681452 11122 consensus_queue.cc:237] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } }
I20260927 02:44:58.684538 10638 catalog_manager.cc:5495] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957628ef1440486b9716014880d56b61 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "957628ef1440486b9716014880d56b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957628ef1440486b9716014880d56b61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46331 } health_report { overall_health: HEALTHY } } }
I20260927 02:44:58.716871 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.713152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.0.98.2:46331)
0927 02:44:58.713285 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:58.713290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:58.713305 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:44:58.713392 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:58.713394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:58.713448 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:58.713544 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:58.713859 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.714350 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.715214 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:58.715429 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:58.715512 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:58.715901 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:58.716310 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:58.716400 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:58.716556 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:44:58.716655 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:44:58.722878 11051 raft_consensus.cc:1261] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 1 FOLLOWER]: Refusing update from remote peer 957628ef1440486b9716014880d56b61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:58.722900 10783 raft_consensus.cc:1261] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [term 1 FOLLOWER]: Refusing update from remote peer 957628ef1440486b9716014880d56b61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:58.723587 11124 consensus_queue.cc:1035] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bbc7a0596974890a26f00d8e1fee9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:58.723775 11122 consensus_queue.cc:1035] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d7cce947e44d2f95ba57d49e10a40a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:58.731257 11128 mvcc.cc:205] Tried to move back new op lower bound from 7333794196361224192 to 7333794196201476096. Current Snapshot: MvccSnapshot[applied={T|T < 7333794196361224192}]
I20260927 02:44:58.733604 11129 mvcc.cc:205] Tried to move back new op lower bound from 7333794196361224192 to 7333794196201476096. Current Snapshot: MvccSnapshot[applied={T|T < 7333794196361224192}]
I20260927 02:44:58.739046 10636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:58.739213 10636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:58.741384 10636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf1cd67bf4ca4bd6a2456403f3a7e6f9: Sending DeleteTablet for 3 replicas of tablet a156627a5523466d9b486c26ec7a1dbe
I20260927 02:44:58.742071 10763 tablet_service.cc:1514] Processing DeleteTablet for tablet a156627a5523466d9b486c26ec7a1dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55656
I20260927 02:44:58.742095 11031 tablet_service.cc:1514] Processing DeleteTablet for tablet a156627a5523466d9b486c26ec7a1dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36136
I20260927 02:44:58.742249 10897 tablet_service.cc:1514] Processing DeleteTablet for tablet a156627a5523466d9b486c26ec7a1dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37740
I20260927 02:44:58.742550 11137 tablet_replica.cc:330] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: stopping tablet replica
I20260927 02:44:58.742811 11136 tablet_replica.cc:330] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc: stopping tablet replica
I20260927 02:44:58.742902 11137 raft_consensus.cc:2229] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:58.743088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 10700
I20260927 02:44:58.743089 11136 raft_consensus.cc:2229] T a156627a5523466d9b486c26ec7a1dbe P 5bbc7a0596974890a26f00d8e1fee9bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:58.743268 11137 raft_consensus.cc:2258] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:58.743790 11138 tablet_replica.cc:330] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: stopping tablet replica
I20260927 02:44:58.744259 11138 raft_consensus.cc:2229] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:58.744593 11138 raft_consensus.cc:2258] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:58.745854 11138 ts_tablet_manager.cc:1905] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:58.746111 11137 ts_tablet_manager.cc:1905] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:58.748581 11138 ts_tablet_manager.cc:1918] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:58.748581 11137 ts_tablet_manager.cc:1918] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:58.748679 11137 log.cc:1196] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/a156627a5523466d9b486c26ec7a1dbe
I20260927 02:44:58.748679 11138 log.cc:1196] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/a156627a5523466d9b486c26ec7a1dbe
I20260927 02:44:58.749002 11138 ts_tablet_manager.cc:1939] T a156627a5523466d9b486c26ec7a1dbe P 957628ef1440486b9716014880d56b61: Deleting consensus metadata
I20260927 02:44:58.749002 11137 ts_tablet_manager.cc:1939] T a156627a5523466d9b486c26ec7a1dbe P 58d7cce947e44d2f95ba57d49e10a40a: Deleting consensus metadata
I20260927 02:44:58.750065 10622 catalog_manager.cc:4841] TS 58d7cce947e44d2f95ba57d49e10a40a (127.0.98.3:35015): tablet a156627a5523466d9b486c26ec7a1dbe (table test-table [id=53991bd063f949e2876386080616e315]) successfully deleted
I20260927 02:44:58.750065 10625 catalog_manager.cc:4841] TS 957628ef1440486b9716014880d56b61 (127.0.98.2:46331): tablet a156627a5523466d9b486c26ec7a1dbe (table test-table [id=53991bd063f949e2876386080616e315]) successfully deleted
W20260927 02:44:58.751120 10623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36035 (error 108)
I20260927 02:44:58.752168 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.751836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36035 (local address 127.0.0.1:55664)
0927 02:44:58.752003 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.752075 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 02:44:58.752329 10623 catalog_manager.cc:4584] TS 5bbc7a0596974890a26f00d8e1fee9bc (127.0.98.1:36035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a156627a5523466d9b486c26ec7a1dbe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36035: connect: Connection refused (error 111)
I20260927 02:44:58.756068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 10834
I20260927 02:44:58.761476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 10968
I20260927 02:44:58.766966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 10606
2026-09-27T02:44:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2350 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[11142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[11142](info): set up 2 sockets
Sep 27 02:44:58 dist-test-slave-gkrl krb5kdc[11142](info): commencing operation
krb5kdc: starting...
W20260927 02:44:59.796931 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:59 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:44:59.967499 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8LsHFz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:45:02.597 UTC [11172] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:45:02.597 UTC [11172] LOG: listening on IPv4 address "127.0.98.20", port 58725
2026-09-27 02:45:02.601 UTC [11172] LOG: listening on Unix socket "/tmp/.s.PGSQL.58725"
2026-09-27 02:45:02.635 UTC [11175] LOG: database system was shut down at 2026-09-27 02:45:01 UTC
2026-09-27 02:45:02.639 UTC [11172] LOG: database system is ready to accept connections
I20260927 02:45:03.600845 392 mini_postgres.cc:96] Postgres bound to 58725
2026-09-27 02:45:03.607 UTC [11183] FATAL: database "slave" does not exist
127.0.98.20:58725 - accepting connections
I20260927 02:45:03.607735 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:45:03.622315 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:45:04.114691 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:45:04.114835 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-admin
2026-09-27 02:45:04,394 [I] DB FLAVOR :POSTGRES
2026-09-27 02:45:04,395 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:45:04,395 [I] Checking connection..
2026-09-27 02:45:04,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:45:04,672 [I] Checking connection passed.
2026-09-27 02:45:04,672 [I] --------- Verifying version history table ---------
2026-09-27 02:45:04,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:45:04,918 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:45:04,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:45:05,181 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:45:05,181 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:45:05,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:45:05.417 UTC [11247] WARNING: there is no transaction in progress
2026-09-27 02:45:05,423 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:45:05,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:45:05,664 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:45:05,664 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:45:05,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:45:05,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:45:06,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gkrl.c.gcp-upstream.internal','N') ;"
2026-09-27 02:45:06,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:45:06,632 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:45:06,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:45:06,889 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:45:06,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:45:07,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:45:07,384 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:45:07,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:45:08.323 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.339 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.356 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.370 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.387 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.405 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.459 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.466 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.480 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.490 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.500 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.513 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.522 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.529 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.536 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:08.542 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 02:45:09,189 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:45:09,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:45:09,431 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:45:09,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:45:09,670 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:45:09,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:45:09,897 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:45:09,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:45:10,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:45:10,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gkrl.c.gcp-upstream.internal';"
2026-09-27 02:45:10,622 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:45:10,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:45:10,861 [I] DB_PATCHES have already been applied
I20260927 02:45:10.867219 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8LsHFz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:45:10.867280 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:45:10.870713 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:45461
Sep 27, 2026 2:45:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:45:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:45:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:45:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:45:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:45:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:45:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:45:12.014528 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.407s user 0.000s sys 0.010s
Sep 27, 2026 2:45:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:45:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:45:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:45:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:45:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:45:33.468653 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:45:33 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477133, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:45:33.550447 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:45:33.559973 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:45:34.454399 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:45:34.454496 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms
I20260927 02:45:34.454589 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58725
2026-09-27 02:45:34,636 [I] DB FLAVOR :POSTGRES
2026-09-27 02:45:34,636 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:45:34,637 [I] Checking connection
2026-09-27 02:45:34,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:45:34,896 [I] connection success
2026-09-27 02:45:34,896 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:45:34,896 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:45:34,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:45:35,122 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:45:35,122 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:45:35,122 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:45:35,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:45:35,374 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:45:35.841549 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:45:36.062444 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:45:36.102240 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:45:36 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477133, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:45:36.462966 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:45:36.463129 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:45:36.463158 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:45:36.467267 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11535
I20260927 02:45:36.467393 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51699
Sep 27, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:45:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:45:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:45:38.806587 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.256s user 0.008s sys 0.157s
I20260927 02:45:38.806735 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:45:38.806854 392 mini_ranger_kms.cc:401] 127.0.98.20:51699/kms/v1/keys
Sep 27 02:45:38 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477138, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:45:39 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477138, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:45:39 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477138, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:45:41 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:45:41.974834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44667
--ranger_config_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51699/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:42.092133 11586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:42.092432 11586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:42.092541 11586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:42.096833 11586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:45:42.096916 11586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:42.096953 11586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:42.097014 11586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:45:42.097050 11586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:45:42.102622 11586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51699/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44667
--ranger_config_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:45:42.103667 11586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:42.104748 11586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:42.114712 11592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:42.114883 11586 server_base.cc:1034] running on GCE node
W20260927 02:45:42.114780 11594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:42.114717 11591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:42.115688 11586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:42.116899 11586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:42.118081 11586 hybrid_clock.cc:648] HybridClock initialized: now 1790477142118063 us; error 35 us; skew 500 ppm
Sep 27 02:45:42 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:42.120966 11586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:45:42.121975 11586 webserver.cc:460] Webserver started at http://127.0.98.62:33027/ using document root <none> and password file <none>
I20260927 02:45:42.122550 11586 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:42.122606 11586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:42.122845 11586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:45:42 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:45:42.184489 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "473ed184bcf441658d6df7180e1d611d"
format_stamp: "Formatted at 2026-09-27 02:45:42 on dist-test-slave-gkrl"
server_key: "ea36e4b62c047f3dc7684b33f316d598"
server_key_iv: "e199f5e048da7a8fd7337b49cc19b9fa"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:42.185227 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "473ed184bcf441658d6df7180e1d611d"
format_stamp: "Formatted at 2026-09-27 02:45:42 on dist-test-slave-gkrl"
server_key: "ea36e4b62c047f3dc7684b33f316d598"
server_key_iv: "e199f5e048da7a8fd7337b49cc19b9fa"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:42.190135 11586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 02:45:42.206714 11601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:42.208477 11586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:45:42.208626 11586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "473ed184bcf441658d6df7180e1d611d"
format_stamp: "Formatted at 2026-09-27 02:45:42 on dist-test-slave-gkrl"
server_key: "ea36e4b62c047f3dc7684b33f316d598"
server_key_iv: "e199f5e048da7a8fd7337b49cc19b9fa"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:42.208775 11586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:42.237767 11586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:42.240906 11586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:42.241118 11586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:42.249064 11586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44667
I20260927 02:45:42.249128 11653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44667 every 8 connection(s)
I20260927 02:45:42.250059 11586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:45:42.252700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 11586
I20260927 02:45:42.252848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:45:42.253167 392 external_mini_cluster.cc:1428] Setting key c01cce9c062e5517ed426119d93cffb2
I20260927 02:45:42.253471 11654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:42.260850 11654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d: Bootstrap starting.
Sep 27 02:45:42 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:42.263494 11654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:42.264777 11654 log.cc:824] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:42.267011 11654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d: No bootstrap required, opened a new log
I20260927 02:45:42.267795 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:42.254586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.0.98.62:44667)
0927 02:45:42.254987 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:42.255008 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:42.255251 (+ 243us) server_negotiation.cc:407] Connection header received
0927 02:45:42.256380 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:42.256399 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:42.256750 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:42.257093 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:42.257848 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:42.258829 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:42.259684 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:42.260244 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:42.260428 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:42.262923 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:42.262928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:42.262943 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:42.262968 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:42.264673 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:42.265159 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:42.265166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:42.265175 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:42.265306 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:42.265632 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:42.265639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:42.265643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:42.266111 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:42.266280 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:42.266602 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 02:45:42.266862 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":138,"threads_started":1}
I20260927 02:45:42.270547 11654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473ed184bcf441658d6df7180e1d611d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44667 } }
I20260927 02:45:42.270752 11654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:42.270809 11654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473ed184bcf441658d6df7180e1d611d, State: Initialized, Role: FOLLOWER
I20260927 02:45:42.271245 11654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473ed184bcf441658d6df7180e1d611d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44667 } }
I20260927 02:45:42.271391 11654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:45:42.271502 11654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:45:42.271636 11654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:42.272614 11654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473ed184bcf441658d6df7180e1d611d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44667 } }
I20260927 02:45:42.272950 11654 leader_election.cc:302] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 473ed184bcf441658d6df7180e1d611d; no voters:
I20260927 02:45:42.273253 11654 leader_election.cc:288] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:45:42.273365 11659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:45:42.273589 11659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [term 1 LEADER]: Becoming Leader. State: Replica: 473ed184bcf441658d6df7180e1d611d, State: Running, Role: LEADER
I20260927 02:45:42.273941 11659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473ed184bcf441658d6df7180e1d611d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44667 } }
I20260927 02:45:42.274449 11654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:45:42.275224 11659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 473ed184bcf441658d6df7180e1d611d. Latest consensus state: current_term: 1 leader_uuid: "473ed184bcf441658d6df7180e1d611d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473ed184bcf441658d6df7180e1d611d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44667 } } }
I20260927 02:45:42.275411 11659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [sys.catalog]: This master's current role is: LEADER
I20260927 02:45:42.275686 11660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "473ed184bcf441658d6df7180e1d611d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473ed184bcf441658d6df7180e1d611d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44667 } } }
I20260927 02:45:42.275808 11668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:45:42.275843 11660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d [sys.catalog]: This master's current role is: LEADER
I20260927 02:45:42.279673 11668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:45:42.280203 11654 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:45:42.290114 11668 catalog_manager.cc:1347] Generated new cluster ID: 36f7f466f43449e9ab0f502d95f6b78d
I20260927 02:45:42.290194 11668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:45:42.300741 11668 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:45:42.302055 11668 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:45:42.308512 11668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d: Generated new TSK 0
I20260927 02:45:42.309376 11668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:45:43.007619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44667
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:43.125650 11699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:43.125926 11699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:43.126019 11699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:43.130326 11699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:43.130412 11699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:43.130499 11699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:45:43.136204 11699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:45:43.137313 11699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:43.138322 11699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:43.147661 11707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:43.147665 11705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:43.147912 11699 server_base.cc:1034] running on GCE node
W20260927 02:45:43.147711 11704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:43.148444 11699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:43.149077 11699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:43.150264 11699 hybrid_clock.cc:648] HybridClock initialized: now 1790477143150244 us; error 39 us; skew 500 ppm
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:43.153399 11699 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:45:43.154361 11699 webserver.cc:460] Webserver started at http://127.0.98.1:45395/ using document root <none> and password file <none>
I20260927 02:45:43.154906 11699 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:43.154994 11699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:43.155237 11699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:45:43.170562 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "bc74eae6a2af44ad811206269601a27a"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "cb846e6449f055397a12ee6190112019"
server_key_iv: "6de165db8160812ef828810066d02da9"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.171240 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bc74eae6a2af44ad811206269601a27a"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "cb846e6449f055397a12ee6190112019"
server_key_iv: "6de165db8160812ef828810066d02da9"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.181321 11699 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260927 02:45:43.198258 11716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.199807 11699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 02:45:43.199960 11699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "bc74eae6a2af44ad811206269601a27a"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "cb846e6449f055397a12ee6190112019"
server_key_iv: "6de165db8160812ef828810066d02da9"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.200107 11699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:43.222666 11699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:43.226413 11699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:43.226617 11699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:43.227310 11699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:43.228322 11699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:43.228384 11699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.228435 11699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:43.228462 11699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.239499 11699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43937
I20260927 02:45:43.240681 11699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:45:43.245097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 11699
I20260927 02:45:43.245220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:45:43.245523 392 external_mini_cluster.cc:1428] Setting key e1ae444e63da7f135038c44bba3b0a33
I20260927 02:45:43.249293 11829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43937 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:45:43.281647 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:43.251461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44667 (local address 127.0.98.1:54279)
0927 02:45:43.258973 (+ 7512us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:43.259035 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:43.260090 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:43.264256 (+ 4166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:43.264268 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:43.264653 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:43.265396 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:43.265423 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:43.266852 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:43.266857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:43.268488 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:43.268509 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:43.269280 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:43.269286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:43.269534 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:43.270268 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:43.270294 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:43.273909 (+ 3615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:43.278197 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:43.278205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:43.278225 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:43.278617 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:43.278937 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:43.278940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:43.278942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:43.279132 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:43.279640 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:43.279647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:43.280013 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 02:45:43.280215 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 02:45:43.280556 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7271,"thread_start_us":125,"threads_started":1}
I20260927 02:45:43.282505 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:43.251449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54279 (local address 127.0.98.62:44667)
0927 02:45:43.263755 (+ 12306us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:43.263764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:43.263782 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:45:43.263826 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:43.263831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:43.263920 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:43.264032 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:43.265610 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:43.266680 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:43.268645 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:43.269080 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:43.269215 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:43.276021 (+ 6806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:43.276025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:43.276028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:43.276059 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:43.278047 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:43.278739 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:43.278742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:43.278744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:43.278836 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:43.279245 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:43.279251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:43.279254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:43.279483 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:43.279582 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:43.282205 (+ 2623us) server_negotiation.cc:299] Negotiation successful
0927 02:45:43.282335 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12220,"thread_start_us":109,"threads_started":1}
I20260927 02:45:43.283620 11830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44667
I20260927 02:45:43.283890 11830 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:43.284461 11830 heartbeater.cc:511] Master 127.0.98.62:44667 requested a full tablet report, sending...
I20260927 02:45:43.286247 11618 ts_manager.cc:194] Registered new tserver with Master: bc74eae6a2af44ad811206269601a27a (127.0.98.1:43937)
I20260927 02:45:43.287772 11618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:45:43.310700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44667
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 02:45:43.451192 11838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:43.451470 11838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:43.451543 11838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:43.455804 11838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:43.455916 11838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:43.456025 11838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:45:43.461476 11838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:45:43.462486 11838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:43.463552 11838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:43.473428 11845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:43.473711 11847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:43.473399 11844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:43.473789 11838 server_base.cc:1034] running on GCE node
I20260927 02:45:43.474316 11838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:43.475090 11838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:43.476285 11838 hybrid_clock.cc:648] HybridClock initialized: now 1790477143476224 us; error 77 us; skew 500 ppm
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:43.479290 11838 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:45:43.480295 11838 webserver.cc:460] Webserver started at http://127.0.98.2:41775/ using document root <none> and password file <none>
I20260927 02:45:43.480857 11838 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:43.480945 11838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:43.481158 11838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:45:43.497364 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "d7116fe87df74add814120957ea5838a"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "e6ce1b4185cb58316c4f564860450a35"
server_key_iv: "899b00dd700082f5256fbf78eaaf81f5"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.498401 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d7116fe87df74add814120957ea5838a"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "e6ce1b4185cb58316c4f564860450a35"
server_key_iv: "899b00dd700082f5256fbf78eaaf81f5"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.505388 11838 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.003s
I20260927 02:45:43.522845 11854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.524546 11838 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 02:45:43.524684 11838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "d7116fe87df74add814120957ea5838a"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "e6ce1b4185cb58316c4f564860450a35"
server_key_iv: "899b00dd700082f5256fbf78eaaf81f5"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.524822 11838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:43.549394 11838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:43.553074 11838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:43.553261 11838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:43.553913 11838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:43.554910 11838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:43.554982 11838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.555040 11838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:43.555071 11838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.568990 11838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41625
I20260927 02:45:43.570078 11838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:45:43.571123 11967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41625 every 8 connection(s)
I20260927 02:45:43.579350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 11838
I20260927 02:45:43.579505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:45:43.579845 392 external_mini_cluster.cc:1428] Setting key cce4316bafe1721b46657c624a6f201f
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:43.607678 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:43.574659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37277 (local address 127.0.98.62:44667)
0927 02:45:43.574802 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:43.574808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:43.580692 (+ 5884us) server_negotiation.cc:407] Connection header received
0927 02:45:43.587481 (+ 6789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:43.587485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:43.587547 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:43.587608 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:43.589466 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:43.590320 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:43.593730 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:43.594107 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:43.594196 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:43.602508 (+ 8312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:43.602513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:43.602516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:43.602550 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:43.604388 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:43.605195 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:43.605200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:43.605202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:43.605292 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:43.605745 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:43.605749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:43.605750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:43.606012 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:43.606108 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:43.606904 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 02:45:43.607492 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:45:43.609735 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:43.573605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44667 (local address 127.0.98.2:37277)
0927 02:45:43.580544 (+ 6939us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:43.580606 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:43.581606 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:43.588101 (+ 6495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:43.588112 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:43.588510 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:43.589263 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:43.589285 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:43.590502 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:43.590506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:43.591944 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:43.591955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:43.594410 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:43.594415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:43.594647 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:43.600462 (+ 5815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:43.600489 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:43.602261 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:43.604561 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:43.604567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:43.604587 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:43.605066 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:43.605406 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:43.605409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:43.605411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:43.605623 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:43.606207 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:43.606213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:43.606593 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 02:45:43.608212 (+ 1619us) client_negotiation.cc:240] Negotiation successful
0927 02:45:43.608599 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6683,"thread_start_us":139,"threads_started":1}
I20260927 02:45:43.611402 11968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44667
I20260927 02:45:43.611673 11968 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:43.612310 11968 heartbeater.cc:511] Master 127.0.98.62:44667 requested a full tablet report, sending...
I20260927 02:45:43.613552 11618 ts_manager.cc:194] Registered new tserver with Master: d7116fe87df74add814120957ea5838a (127.0.98.2:41625)
I20260927 02:45:43.614352 11618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:45:43.640188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44667
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:43.815289 11973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:43.815732 11973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:43.815853 11973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:43.821532 11973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:43.821787 11973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:43.821928 11973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:45:43.829514 11973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:45:43.830930 11973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:43.832260 11973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:43.844372 11981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:43.844422 11979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:43.845769 11973 server_base.cc:1034] running on GCE node
W20260927 02:45:43.849519 11978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:43.851148 11973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:43.851943 11973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:43.855208 11973 hybrid_clock.cc:648] HybridClock initialized: now 1790477143855132 us; error 84 us; skew 500 ppm
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:43.858794 11973 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:45:43.859820 11973 webserver.cc:460] Webserver started at http://127.0.98.3:40593/ using document root <none> and password file <none>
I20260927 02:45:43.860536 11973 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:43.860607 11973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:43.860818 11973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:45:43 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:45:43.883077 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f9eab80ed2849a79990ac4b78887753"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "e4f9c5e20f5c3c94f8fdfc16250e17c2"
server_key_iv: "31ce8d624bd700a5b05f8ff4790a5b23"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.883813 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f9eab80ed2849a79990ac4b78887753"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "e4f9c5e20f5c3c94f8fdfc16250e17c2"
server_key_iv: "31ce8d624bd700a5b05f8ff4790a5b23"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.923564 11973 fs_manager.cc:691] Time spent creating directory manager: real 0.039s user 0.002s sys 0.002s
I20260927 02:45:43.950457 11988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.952168 11973 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 02:45:43.952302 11973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "0f9eab80ed2849a79990ac4b78887753"
format_stamp: "Formatted at 2026-09-27 02:45:43 on dist-test-slave-gkrl"
server_key: "e4f9c5e20f5c3c94f8fdfc16250e17c2"
server_key_iv: "31ce8d624bd700a5b05f8ff4790a5b23"
server_key_version: "kuduclusterkey@0"
I20260927 02:45:43.952441 11973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:43.980423 11973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:43.984149 11973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:43.984329 11973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:43.985008 11973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:43.986047 11973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:43.986114 11973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.986177 11973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:43.986212 11973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:43.996236 11973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33321
I20260927 02:45:43.997285 11973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:45:44.006065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 11973
I20260927 02:45:44.006227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:45:44.006552 392 external_mini_cluster.cc:1428] Setting key ced3efc8257616bed2d7d63c0f243de8
I20260927 02:45:44.007819 12101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33321 every 8 connection(s)
Sep 27 02:45:44 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:44.028661 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.012029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47001 (local address 127.0.98.62:44667)
0927 02:45:44.012149 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:44.012153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:44.013157 (+ 1004us) server_negotiation.cc:407] Connection header received
0927 02:45:44.014431 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:44.014434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:44.014491 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:44.014552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:44.016231 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.016914 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.018523 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.018861 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.018930 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.021935 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:44.021938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:44.021942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:44.021969 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:44.025093 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:44.026192 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:44.026196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:44.026198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:44.026298 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:44.026772 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:44.026776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:44.026778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:44.027042 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:44.027133 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:44.027926 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 02:45:44.028076 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:45:44.030647 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.011013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44667 (local address 127.0.98.3:47001)
0927 02:45:44.013001 (+ 1988us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:44.013065 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:44.014168 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:44.014976 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:44.014986 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:44.015349 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:44.016070 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.016090 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.017071 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.017074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.018395 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.018404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.019118 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.019121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.019304 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.020044 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:44.020065 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:44.021745 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:44.025248 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:44.025255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:44.025277 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:44.026066 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:44.026421 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:44.026425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:44.026428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:44.026647 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:44.027231 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:44.027238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:44.027614 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 02:45:44.028887 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0927 02:45:44.029353 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1752,"thread_start_us":161,"threads_started":1}
I20260927 02:45:44.032346 12102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44667
I20260927 02:45:44.032615 12102 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:44.033167 12102 heartbeater.cc:511] Master 127.0.98.62:44667 requested a full tablet report, sending...
I20260927 02:45:44.034267 11618 ts_manager.cc:194] Registered new tserver with Master: 0f9eab80ed2849a79990ac4b78887753 (127.0.98.3:33321)
I20260927 02:45:44.034996 11618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47001
I20260927 02:45:44.036584 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:45:44.048027 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.038795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.62:44667)
0927 02:45:44.038897 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:44.038901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:44.039406 (+ 505us) server_negotiation.cc:407] Connection header received
0927 02:45:44.039566 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:44.039569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:44.039611 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:44.039657 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:44.040577 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.041343 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.042517 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.042865 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.042934 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.044640 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:44.044644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:44.044647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:44.044671 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:44.046295 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:44.046751 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:44.046755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:44.046757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:44.046830 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:44.047119 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:44.047122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:44.047129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:44.047330 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:44.047426 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:44.047752 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 02:45:44.047855 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:45:44.051122 11618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:45:44.057556 11618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:45:44.086093 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.068676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33321 (local address 127.0.0.1:51126)
0927 02:45:44.073330 (+ 4654us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:44.073378 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:44.073517 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:44.074252 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:44.074258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:44.074279 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:44.074718 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.074723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.077049 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.077053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.080838 (+ 3785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.080845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.085536 (+ 4691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.085540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.085667 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.085712 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:45:44.085817 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 02:45:44.085934 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4483,"thread_start_us":90,"threads_started":1}
I20260927 02:45:44.087390 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.068997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.0.98.3:33321)
0927 02:45:44.069406 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:44.069417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:44.073663 (+ 4246us) server_negotiation.cc:407] Connection header received
0927 02:45:44.073741 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:44.073746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:44.073951 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:44.074088 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:44.074833 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.076916 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.080988 (+ 4072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.081789 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.082614 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.082774 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:44.087110 (+ 4336us) server_negotiation.cc:299] Negotiation successful
0927 02:45:44.087249 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":98,"threads_started":1}
I20260927 02:45:44.089212 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.070810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41625 (local address 127.0.0.1:59966)
0927 02:45:44.074443 (+ 3633us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:44.074456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:44.074530 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:44.080249 (+ 5719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:44.080252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:44.080268 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:44.080362 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.080367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.084164 (+ 3797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.084167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.085403 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.085413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.088824 (+ 3411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.088828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.088932 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.088955 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:45:44.089024 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:45:44.089089 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3575,"thread_start_us":87,"threads_started":1}
I20260927 02:45:44.089718 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.071195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.0.98.2:41625)
0927 02:45:44.079673 (+ 8478us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:44.079690 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:44.079711 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:45:44.079778 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:44.079783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:44.079979 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:44.080118 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:44.082839 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.084034 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.087648 (+ 3614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.088517 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.088630 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.088765 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:44.089475 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 02:45:44.089568 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8361,"thread_start_us":104,"threads_started":1}
I20260927 02:45:44.090732 12036 tablet_service.cc:1467] Processing CreateTablet for tablet 92442b17a0fa4bfab8698446cd32396e (DEFAULT_TABLE table=test-table [id=457c6c9f719e49e780909dd3385aa7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:44.091897 12036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92442b17a0fa4bfab8698446cd32396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:44.092224 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.071642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43937 (local address 127.0.0.1:49254)
0927 02:45:44.076084 (+ 4442us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:44.076098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:44.076189 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:44.079441 (+ 3252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:44.079444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:44.079463 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:44.079557 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.079563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.089649 (+ 10086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.089653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.090801 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.090811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.091835 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.091839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.091964 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.091989 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:45:44.092040 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:45:44.092103 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4378,"thread_start_us":80,"threads_started":1}
I20260927 02:45:44.092505 11902 tablet_service.cc:1467] Processing CreateTablet for tablet 92442b17a0fa4bfab8698446cd32396e (DEFAULT_TABLE table=test-table [id=457c6c9f719e49e780909dd3385aa7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:44.092978 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.072042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.0.98.1:43937)
0927 02:45:44.078948 (+ 6906us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:44.078955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:44.078977 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:45:44.079049 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:44.079054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:44.079251 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:44.079400 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:44.088351 (+ 8951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.089502 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.090965 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.091698 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.092549 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.092677 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:44.092725 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:45:44.092807 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6801,"thread_start_us":85,"threads_started":1}
I20260927 02:45:44.093521 11902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92442b17a0fa4bfab8698446cd32396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:44.095499 11762 tablet_service.cc:1467] Processing CreateTablet for tablet 92442b17a0fa4bfab8698446cd32396e (DEFAULT_TABLE table=test-table [id=457c6c9f719e49e780909dd3385aa7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:44.096486 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92442b17a0fa4bfab8698446cd32396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:44.109189 12122 tablet_bootstrap.cc:492] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: Bootstrap starting.
I20260927 02:45:44.110862 12121 tablet_bootstrap.cc:492] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Bootstrap starting.
I20260927 02:45:44.112340 12122 tablet_bootstrap.cc:654] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:44.113163 12122 log.cc:824] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:44.114158 12121 tablet_bootstrap.cc:654] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:44.115566 12122 tablet_bootstrap.cc:492] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: No bootstrap required, opened a new log
I20260927 02:45:44.115764 12122 ts_tablet_manager.cc:1397] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 02:45:44.117075 12121 log.cc:824] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:44.119601 12122 raft_consensus.cc:348] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.119889 12122 raft_consensus.cc:374] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:44.119956 12122 raft_consensus.cc:729] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9eab80ed2849a79990ac4b78887753, State: Initialized, Role: FOLLOWER
I20260927 02:45:44.120443 12122 consensus_queue.cc:260] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.120832 12123 tablet_bootstrap.cc:492] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a: Bootstrap starting.
I20260927 02:45:44.121449 12122 ts_tablet_manager.cc:1428] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:45:44.121795 12102 heartbeater.cc:503] Master 127.0.98.62:44667 was elected leader, sending a full tablet report...
I20260927 02:45:44.126926 12121 tablet_bootstrap.cc:492] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: No bootstrap required, opened a new log
I20260927 02:45:44.127156 12121 ts_tablet_manager.cc:1397] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20260927 02:45:44.127737 12123 tablet_bootstrap.cc:654] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:44.128667 12123 log.cc:824] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:44.133630 12121 raft_consensus.cc:348] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.133921 12121 raft_consensus.cc:374] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:44.133988 12121 raft_consensus.cc:729] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7116fe87df74add814120957ea5838a, State: Initialized, Role: FOLLOWER
I20260927 02:45:44.134487 12121 consensus_queue.cc:260] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.135552 12121 ts_tablet_manager.cc:1428] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260927 02:45:44.135954 11968 heartbeater.cc:503] Master 127.0.98.62:44667 was elected leader, sending a full tablet report...
I20260927 02:45:44.140383 12123 tablet_bootstrap.cc:492] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a: No bootstrap required, opened a new log
I20260927 02:45:44.140597 12123 ts_tablet_manager.cc:1397] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260927 02:45:44.145211 12123 raft_consensus.cc:348] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.145462 12123 raft_consensus.cc:374] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:44.145505 12123 raft_consensus.cc:729] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc74eae6a2af44ad811206269601a27a, State: Initialized, Role: FOLLOWER
I20260927 02:45:44.146107 12123 consensus_queue.cc:260] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.147181 12123 ts_tablet_manager.cc:1428] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 02:45:44.147547 11830 heartbeater.cc:503] Master 127.0.98.62:44667 was elected leader, sending a full tablet report...
Sep 27 02:45:44 dist-test-slave-gkrl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 02:45:44.257341 11831 tablet.cc:2369] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:45:44.259150 12103 tablet.cc:2369] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:45:44.322224 11969 tablet.cc:2369] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:45:44.384305 12126 raft_consensus.cc:482] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:45:44.384558 12126 raft_consensus.cc:504] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.385947 12126 leader_election.cc:288] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7116fe87df74add814120957ea5838a (127.0.98.2:41625), bc74eae6a2af44ad811206269601a27a (127.0.98.1:43937)
I20260927 02:45:44.391417 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.386211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41625 (local address 127.0.98.3:59989)
0927 02:45:44.386433 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:44.386450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:44.386562 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:44.387222 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:44.387227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:44.387245 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:44.387419 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.387425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.388574 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.388577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.389918 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.389929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.390893 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.390897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.391051 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.391111 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 02:45:44.391186 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:45:44.391261 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":153,"spinlock_wait_cycles":37760}
I20260927 02:45:44.391402 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.386728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38531 (local address 127.0.98.1:43937)
0927 02:45:44.386858 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:44.386863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:44.386881 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:45:44.387021 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:44.387024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:44.387080 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:44.387123 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:44.387541 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.388348 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.390064 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.390658 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.390746 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.390809 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:44.391230 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 02:45:44.391304 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:45:44.391937 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.386522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59989 (local address 127.0.98.2:41625)
0927 02:45:44.386878 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:44.386883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:44.386897 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:45:44.386952 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:44.386958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:44.387010 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:44.387076 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:44.387557 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.388349 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.390064 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.390676 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.390762 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.390810 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:44.391735 (+ 925us) server_negotiation.cc:299] Negotiation successful
0927 02:45:44.391806 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260927 02:45:44.392457 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.386289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43937 (local address 127.0.98.3:38531)
0927 02:45:44.386794 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:44.386808 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:44.386886 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:44.387257 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:44.387259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:44.387281 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:44.387413 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.387419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.388494 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.388497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.389921 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:44.389929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.390893 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.390897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:44.391103 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.391122 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:45:44.391841 (+ 719us) client_negotiation.cc:240] Negotiation successful
0927 02:45:44.391930 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":90,"thread_start_us":106,"threads_started":1}
I20260927 02:45:44.392542 11907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92442b17a0fa4bfab8698446cd32396e" candidate_uuid: "0f9eab80ed2849a79990ac4b78887753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7116fe87df74add814120957ea5838a" is_pre_election: true
I20260927 02:45:44.392867 11907 raft_consensus.cc:2454] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9eab80ed2849a79990ac4b78887753 in term 0.
I20260927 02:45:44.393172 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92442b17a0fa4bfab8698446cd32396e" candidate_uuid: "0f9eab80ed2849a79990ac4b78887753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc74eae6a2af44ad811206269601a27a" is_pre_election: true
I20260927 02:45:44.393357 11990 leader_election.cc:302] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9eab80ed2849a79990ac4b78887753, d7116fe87df74add814120957ea5838a; no voters:
I20260927 02:45:44.393460 11777 raft_consensus.cc:2454] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9eab80ed2849a79990ac4b78887753 in term 0.
I20260927 02:45:44.393649 12126 raft_consensus.cc:2781] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:45:44.393718 12126 raft_consensus.cc:482] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:45:44.393774 12126 raft_consensus.cc:3037] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:44.395103 12126 raft_consensus.cc:504] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.395449 12126 leader_election.cc:288] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [CANDIDATE]: Term 1 election: Requested vote from peers d7116fe87df74add814120957ea5838a (127.0.98.2:41625), bc74eae6a2af44ad811206269601a27a (127.0.98.1:43937)
I20260927 02:45:44.396086 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92442b17a0fa4bfab8698446cd32396e" candidate_uuid: "0f9eab80ed2849a79990ac4b78887753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc74eae6a2af44ad811206269601a27a"
I20260927 02:45:44.396200 11777 raft_consensus.cc:3037] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:44.397392 11777 raft_consensus.cc:2454] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9eab80ed2849a79990ac4b78887753 in term 1.
I20260927 02:45:44.397711 11992 leader_election.cc:302] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9eab80ed2849a79990ac4b78887753, bc74eae6a2af44ad811206269601a27a; no voters:
I20260927 02:45:44.397924 11907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92442b17a0fa4bfab8698446cd32396e" candidate_uuid: "0f9eab80ed2849a79990ac4b78887753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7116fe87df74add814120957ea5838a"
I20260927 02:45:44.398038 11907 raft_consensus.cc:3037] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:44.399221 11907 raft_consensus.cc:2454] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9eab80ed2849a79990ac4b78887753 in term 1.
I20260927 02:45:44.399611 12126 raft_consensus.cc:2781] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:45:44.399837 12126 raft_consensus.cc:686] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 1 LEADER]: Becoming Leader. State: Replica: 0f9eab80ed2849a79990ac4b78887753, State: Running, Role: LEADER
I20260927 02:45:44.400213 12126 consensus_queue.cc:237] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } }
I20260927 02:45:44.406076 11618 catalog_manager.cc:5495] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f9eab80ed2849a79990ac4b78887753 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0f9eab80ed2849a79990ac4b78887753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9eab80ed2849a79990ac4b78887753" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260927 02:45:44.442613 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.437511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.0.98.3:33321)
0927 02:45:44.437798 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:44.437801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:44.437813 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:45:44.437881 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:44.437883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:44.437984 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:44.438101 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:45:44.438366 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.439188 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.440459 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:44.440797 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:44.440898 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:44.441319 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:45:44.441881 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:45:44.442013 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:44.442422 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 02:45:44.442480 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260927 02:45:44.455726 11777 raft_consensus.cc:1261] T 92442b17a0fa4bfab8698446cd32396e P bc74eae6a2af44ad811206269601a27a [term 1 FOLLOWER]: Refusing update from remote peer 0f9eab80ed2849a79990ac4b78887753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:45:44.456615 12126 consensus_queue.cc:1035] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc74eae6a2af44ad811206269601a27a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:45:44.465085 11907 raft_consensus.cc:1261] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 1 FOLLOWER]: Refusing update from remote peer 0f9eab80ed2849a79990ac4b78887753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:45:44.465735 12126 consensus_queue.cc:1035] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7116fe87df74add814120957ea5838a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:45:44.470706 12136 mvcc.cc:205] Tried to move back new op lower bound from 7333794383683043328 to 7333794383466438656. Current Snapshot: MvccSnapshot[applied={T|T < 7333794383683043328}]
I20260927 02:45:44.470916 12137 mvcc.cc:205] Tried to move back new op lower bound from 7333794383683043328 to 7333794383466438656. Current Snapshot: MvccSnapshot[applied={T|T < 7333794383683043328}]
I20260927 02:45:44.484438 11617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:45:44.484624 11617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:45:44.487313 11617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 473ed184bcf441658d6df7180e1d611d: Sending DeleteTablet for 3 replicas of tablet 92442b17a0fa4bfab8698446cd32396e
I20260927 02:45:44.488834 12036 tablet_service.cc:1514] Processing DeleteTablet for tablet 92442b17a0fa4bfab8698446cd32396e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51126
I20260927 02:45:44.489759 11762 tablet_service.cc:1514] Processing DeleteTablet for tablet 92442b17a0fa4bfab8698446cd32396e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49254
I20260927 02:45:44.491539 11902 tablet_service.cc:1514] Processing DeleteTablet for tablet 92442b17a0fa4bfab8698446cd32396e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59966
I20260927 02:45:44.493042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 11699
I20260927 02:45:44.496086 12139 mvcc.cc:205] Tried to move back new op lower bound from 7333794383683043328 to 7333794383466438656. Current Snapshot: MvccSnapshot[applied={T|T < 7333794383683043328}]
I20260927 02:45:44.496331 12147 tablet_replica.cc:330] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: stopping tablet replica
I20260927 02:45:44.496631 12147 raft_consensus.cc:2229] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:45:44.496876 12147 raft_consensus.cc:2258] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:45:44.500312 12147 ts_tablet_manager.cc:1905] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:45:44.502993 12147 ts_tablet_manager.cc:1918] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:45:44.503089 12147 log.cc:1196] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/92442b17a0fa4bfab8698446cd32396e
I20260927 02:45:44.503410 12147 ts_tablet_manager.cc:1939] T 92442b17a0fa4bfab8698446cd32396e P d7116fe87df74add814120957ea5838a: Deleting consensus metadata
I20260927 02:45:44.504601 11603 catalog_manager.cc:4841] TS d7116fe87df74add814120957ea5838a (127.0.98.2:41625): tablet 92442b17a0fa4bfab8698446cd32396e (table test-table [id=457c6c9f719e49e780909dd3385aa7ac]) successfully deleted
I20260927 02:45:44.508145 12144 tablet_replica.cc:330] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: stopping tablet replica
I20260927 02:45:44.508602 12144 raft_consensus.cc:2229] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:45:44.509080 12144 raft_consensus.cc:2258] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:45:44.509186 11605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43937 (error 108)
I20260927 02:45:44.510099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 11838
I20260927 02:45:44.513037 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.512625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43937 (local address 127.0.0.1:49258)
0927 02:45:44.512827 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:44.512903 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260927 02:45:44.513320 11605 catalog_manager.cc:4584] TS bc74eae6a2af44ad811206269601a27a (127.0.98.1:43937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92442b17a0fa4bfab8698446cd32396e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43937: connect: Connection refused (error 111)
I20260927 02:45:44.517040 12144 ts_tablet_manager.cc:1905] T 92442b17a0fa4bfab8698446cd32396e P 0f9eab80ed2849a79990ac4b78887753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:45:44.519150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 11973
I20260927 02:45:44.528987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 11586
I20260927 02:45:44.529068 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:44.528763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33321 (local address 127.0.0.1:51148)
0927 02:45:44.528895 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:44.528959 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:45:44.535171 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:45:44.869720 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:45:44.869889 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:45:45.206046 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:45:45.206 UTC [11172] LOG: received smart shutdown request
2026-09-27 02:45:45.209 UTC [11172] LOG: background worker "logical replication launcher" (PID 11181) exited with exit code 1
2026-09-27 02:45:45.209 UTC [11176] LOG: shutting down
2026-09-27 02:45:45.380 UTC [11172] LOG: database system is shut down
2026-09-27T02:45:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46733 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:45:45 dist-test-slave-gkrl krb5kdc[12164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:45:45 dist-test-slave-gkrl krb5kdc[12164](info): set up 2 sockets
Sep 27 02:45:45 dist-test-slave-gkrl krb5kdc[12164](info): commencing operation
krb5kdc: starting...
W20260927 02:45:46.540073 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:45:46 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:45:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:45:46.710282 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8LsHFz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:45:49.216 UTC [12194] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:45:49.216 UTC [12194] LOG: listening on IPv4 address "127.0.98.20", port 56151
2026-09-27 02:45:49.220 UTC [12194] LOG: listening on Unix socket "/tmp/.s.PGSQL.56151"
2026-09-27 02:45:49.240 UTC [12197] LOG: database system was shut down at 2026-09-27 02:45:47 UTC
2026-09-27 02:45:49.244 UTC [12194] LOG: database system is ready to accept connections
I20260927 02:45:50.199801 392 mini_postgres.cc:96] Postgres bound to 56151
2026-09-27 02:45:50.206 UTC [12205] FATAL: database "slave" does not exist
127.0.98.20:56151 - accepting connections
I20260927 02:45:50.206768 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:45:50.221691 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:45:50.708681 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:45:50.708767 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-admin
2026-09-27 02:45:50,982 [I] DB FLAVOR :POSTGRES
2026-09-27 02:45:50,983 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:45:50,983 [I] Checking connection..
2026-09-27 02:45:50,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:45:51,228 [I] Checking connection passed.
2026-09-27 02:45:51,228 [I] --------- Verifying version history table ---------
2026-09-27 02:45:51,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:45:51,472 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:45:51,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:45:51,731 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:45:51,731 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:45:51,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:45:51.986 UTC [12269] WARNING: there is no transaction in progress
2026-09-27 02:45:51,992 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:45:51,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:45:52,261 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:45:52,261 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:45:52,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:45:52,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:45:52,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gkrl.c.gcp-upstream.internal','N') ;"
2026-09-27 02:45:52,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:45:53,237 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:45:53,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:45:53,496 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:45:53,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:45:53,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:45:53,947 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:45:53,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:45:54.923 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:54.941 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:54.958 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:54.973 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:54.989 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.009 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.067 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.074 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.089 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.099 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.109 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.123 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.133 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.140 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.147 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55.154 UTC [12377] WARNING: there is no transaction in progress
2026-09-27 02:45:55,794 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:45:55,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:45:56,011 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:45:56,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:45:56,237 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:45:56,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:45:56,473 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:45:56,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:45:56,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:45:56,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gkrl.c.gcp-upstream.internal';"
2026-09-27 02:45:57,184 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:45:57,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:45:57,442 [I] DB_PATCHES have already been applied
I20260927 02:45:57.447600 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8LsHFz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:45:57.447700 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:45:57.451009 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:41945
Sep 27, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:45:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:45:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:45:58.565834 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.359s user 0.000s sys 0.010s
Sep 27, 2026 2:46:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:46:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 2:46:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:46:18 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:46:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:46:20.120000 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:46:20 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477180, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:46:20.225099 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:46:20.234838 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:46:21.112489 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:46:21.112562 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms
I20260927 02:46:21.112586 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56151
2026-09-27 02:46:21,278 [I] DB FLAVOR :POSTGRES
2026-09-27 02:46:21,278 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:46:21,279 [I] Checking connection
2026-09-27 02:46:21,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:46:21,518 [I] connection success
2026-09-27 02:46:21,518 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:46:21,518 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:46:21,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:46:21,740 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:46:21,740 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:46:21,740 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:46:21,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:46:21,988 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:46:22.413089 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:46:22.595510 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:46:22.638417 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:46:22 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477180, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:23.021451 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:46:23.021589 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task8LsHFz/build/debug/bin/postgresql.jar:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8LsHFz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:46:23.021610 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:46:23.025683 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12557
I20260927 02:46:23.025766 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:53361
Sep 27, 2026 2:46:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:46:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:46:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:46:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:46:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:46:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:46:25.281924 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.057s user 0.007s sys 0.144s
I20260927 02:46:25.282060 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:46:25.282138 392 mini_ranger_kms.cc:401] 127.0.98.20:53361/kms/v1/keys
Sep 27 02:46:25 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477185, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:46:25 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477185, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:46:26 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477185, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:46:28 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:28.551396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33965
--ranger_config_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53361/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:28.659406 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:28.659647 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:28.659685 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:28.663584 12608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:28.663638 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:28.663656 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:28.663672 12608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:28.663687 12608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:28.669075 12608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:53361/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33965
--ranger_config_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:28.670006 12608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:28.670918 12608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:28.679610 12614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:28.679611 12616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:28.679705 12608 server_base.cc:1034] running on GCE node
W20260927 02:46:28.679703 12613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:28.680475 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:28.681524 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:28.682725 12608 hybrid_clock.cc:648] HybridClock initialized: now 1790477188682704 us; error 45 us; skew 500 ppm
Sep 27 02:46:28 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:28.685570 12608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:28.686483 12608 webserver.cc:460] Webserver started at http://127.0.98.62:36417/ using document root <none> and password file <none>
I20260927 02:46:28.687083 12608 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:28.687182 12608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:28.687459 12608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:46:28 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:28.743400 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "8c1bd11fc85b4b62a761008ffa0de6af"
format_stamp: "Formatted at 2026-09-27 02:46:28 on dist-test-slave-gkrl"
server_key: "f9329689752c27dbfdaa46c23881ef71"
server_key_iv: "77f2033545d6f2e3570712053604940b"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:28.744083 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c1bd11fc85b4b62a761008ffa0de6af"
format_stamp: "Formatted at 2026-09-27 02:46:28 on dist-test-slave-gkrl"
server_key: "f9329689752c27dbfdaa46c23881ef71"
server_key_iv: "77f2033545d6f2e3570712053604940b"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:28.748234 12608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:46:28.766908 12623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:28.768399 12608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:46:28.768510 12608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "8c1bd11fc85b4b62a761008ffa0de6af"
format_stamp: "Formatted at 2026-09-27 02:46:28 on dist-test-slave-gkrl"
server_key: "f9329689752c27dbfdaa46c23881ef71"
server_key_iv: "77f2033545d6f2e3570712053604940b"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:28.768666 12608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:28.785523 12608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:28.788614 12608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:28.788826 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:28.796903 12608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33965
I20260927 02:46:28.796904 12675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33965 every 8 connection(s)
I20260927 02:46:28.797855 12608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:28.798630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 12608
I20260927 02:46:28.798761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:28.799072 392 external_mini_cluster.cc:1428] Setting key d318bca35f060df1d7806ce812abc55b
I20260927 02:46:28.801247 12676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:28.807494 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Bootstrap starting.
Sep 27 02:46:28 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:28.810209 12676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:28.811162 12676 log.cc:824] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:28.813529 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: No bootstrap required, opened a new log
I20260927 02:46:28.816473 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:28.800499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.0.98.62:33965)
0927 02:46:28.800966 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:28.800985 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:28.801030 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:46:28.801935 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:28.801960 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:28.802400 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:28.802778 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:28.803685 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:28.805300 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:28.806560 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:28.807280 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:28.807439 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:28.810532 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:28.810539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:28.810555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:28.810584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:28.813270 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:28.813783 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:28.813788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:28.813794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:28.813924 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:28.814257 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:28.814261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:28.814263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:28.814776 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:28.814965 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:28.815271 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 02:46:28.815539 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":147,"threads_started":1}
I20260927 02:46:28.817103 12676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260927 02:46:28.817342 12676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:28.817420 12676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1bd11fc85b4b62a761008ffa0de6af, State: Initialized, Role: FOLLOWER
I20260927 02:46:28.817878 12676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260927 02:46:28.818025 12676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:28.818116 12676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:28.818243 12676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:28.819293 12676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260927 02:46:28.819625 12676 leader_election.cc:302] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c1bd11fc85b4b62a761008ffa0de6af; no voters:
I20260927 02:46:28.819972 12676 leader_election.cc:288] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:28.820155 12681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:28.820366 12681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 1 LEADER]: Becoming Leader. State: Replica: 8c1bd11fc85b4b62a761008ffa0de6af, State: Running, Role: LEADER
I20260927 02:46:28.820724 12681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260927 02:46:28.821295 12676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:28.821898 12681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c1bd11fc85b4b62a761008ffa0de6af. Latest consensus state: current_term: 1 leader_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } } }
I20260927 02:46:28.822086 12681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:28.822389 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } } }
I20260927 02:46:28.822492 12688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:28.822557 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:28.825649 12688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:28.826061 12676 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:46:28.840278 12688 catalog_manager.cc:1347] Generated new cluster ID: 030e4dd3d8d74a3fa920b57b01a0e3da
I20260927 02:46:28.840368 12688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:28.857311 12688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:28.858649 12688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:28.864387 12688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Generated new TSK 0
I20260927 02:46:28.865260 12688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:46:29.551674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33965
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:29.712042 12721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:29.712385 12721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:29.712455 12721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:29.718598 12721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:29.718667 12721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:29.718756 12721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:29.726490 12721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:53361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:29.727907 12721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:29.729249 12721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:29.739559 12731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:29.739679 12728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:29.739581 12729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:29.740216 12721 server_base.cc:1034] running on GCE node
I20260927 02:46:29.740595 12721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:29.741219 12721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:29.742408 12721 hybrid_clock.cc:648] HybridClock initialized: now 1790477189742376 us; error 50 us; skew 500 ppm
Sep 27 02:46:29 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:29.745289 12721 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:29.746307 12721 webserver.cc:460] Webserver started at http://127.0.98.1:32875/ using document root <none> and password file <none>
I20260927 02:46:29.747031 12721 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:29.747169 12721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:29.747426 12721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:46:29 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:29.762632 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea25ee677d384834b8f02c16eb4b1466"
format_stamp: "Formatted at 2026-09-27 02:46:29 on dist-test-slave-gkrl"
server_key: "663e2148cc730903c1f54ea6e4919946"
server_key_iv: "c3be9e2c297997821dbe1096a2461517"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:29.763409 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea25ee677d384834b8f02c16eb4b1466"
format_stamp: "Formatted at 2026-09-27 02:46:29 on dist-test-slave-gkrl"
server_key: "663e2148cc730903c1f54ea6e4919946"
server_key_iv: "c3be9e2c297997821dbe1096a2461517"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:29.769340 12721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 02:46:29.787673 12738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:29.789529 12721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 02:46:29.789672 12721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "ea25ee677d384834b8f02c16eb4b1466"
format_stamp: "Formatted at 2026-09-27 02:46:29 on dist-test-slave-gkrl"
server_key: "663e2148cc730903c1f54ea6e4919946"
server_key_iv: "c3be9e2c297997821dbe1096a2461517"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:29.789810 12721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:29.817407 12721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:29.821194 12721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:29.821434 12721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:29.822244 12721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:29.823949 12721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:29.830163 12721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 02:46:29.830324 12721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:29.830422 12721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:29.843823 12721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32935
I20260927 02:46:29.845052 12721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:29.849402 12851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32935 every 8 connection(s)
I20260927 02:46:29.850831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 12721
I20260927 02:46:29.850996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:46:29.851334 392 external_mini_cluster.cc:1428] Setting key 4c140b62e6592329ebdf648ccebbb36c
Sep 27 02:46:29 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:29.866240 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:29.848035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33965 (local address 127.0.98.1:32941)
0927 02:46:29.850027 (+ 1992us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:29.850083 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:29.851321 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:29.853664 (+ 2343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:29.853677 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:29.854040 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:29.854747 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:29.854768 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:29.855956 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:29.855960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:29.857356 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:29.857367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:29.858121 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:29.858124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:29.858308 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:29.859010 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:29.859031 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:29.860776 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:29.863007 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:29.863014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:29.863030 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:29.863358 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:29.863687 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:29.863690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:29.863692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:29.863862 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:29.864368 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:29.864375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:29.864709 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 02:46:29.864911 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 02:46:29.865225 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1764,"thread_start_us":136,"threads_started":1}
I20260927 02:46:29.866989 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:29.847519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32941 (local address 127.0.98.62:33965)
0927 02:46:29.853099 (+ 5580us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:29.853111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:29.853126 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:46:29.853164 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:29.853166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:29.853209 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:29.853299 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:29.854918 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:29.855785 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:29.857511 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:29.857858 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:29.857959 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:29.860967 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:29.860971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:29.860974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:29.860999 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:29.862873 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:29.863494 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:29.863498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:29.863500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:29.863588 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:29.863997 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:29.864000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:29.864002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:29.864220 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:29.864315 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:29.866678 (+ 2363us) server_negotiation.cc:299] Negotiation successful
0927 02:46:29.866804 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5505,"thread_start_us":101,"threads_started":1}
I20260927 02:46:29.870005 12852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33965
I20260927 02:46:29.870234 12852 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:29.870782 12852 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
I20260927 02:46:29.872586 12640 ts_manager.cc:194] Registered new tserver with Master: ea25ee677d384834b8f02c16eb4b1466 (127.0.98.1:32935)
I20260927 02:46:29.874140 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32941
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:46:29.918757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33965
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:46:29 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 02:46:30.090199 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:30.090687 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:30.090826 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:30.097445 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:30.097604 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:30.097766 12860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:30.106079 12860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:53361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:30.107788 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:30.109217 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:30.121697 12866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:30.121767 12867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:30.122176 12869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:30.122831 12860 server_base.cc:1034] running on GCE node
I20260927 02:46:30.123270 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:30.124064 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:30.125310 12860 hybrid_clock.cc:648] HybridClock initialized: now 1790477190125290 us; error 33 us; skew 500 ppm
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:30.128865 12860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:30.129866 12860 webserver.cc:460] Webserver started at http://127.0.98.2:43557/ using document root <none> and password file <none>
I20260927 02:46:30.130555 12860 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:30.130640 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:30.130851 12860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:30.150815 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c3f8bea28b34642829d8a474ce8b065"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-gkrl"
server_key: "78d00f8b740e46306cd87072ea0339ce"
server_key_iv: "4904bc93fd1d591918c7fa6f9f443613"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:30.151686 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c3f8bea28b34642829d8a474ce8b065"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-gkrl"
server_key: "78d00f8b740e46306cd87072ea0339ce"
server_key_iv: "4904bc93fd1d591918c7fa6f9f443613"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:30.157289 12860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 02:46:30.180331 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:30.182009 12860 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:46:30.182140 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "0c3f8bea28b34642829d8a474ce8b065"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-gkrl"
server_key: "78d00f8b740e46306cd87072ea0339ce"
server_key_iv: "4904bc93fd1d591918c7fa6f9f443613"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:30.182277 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:30.209502 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:30.213214 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:30.213465 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:30.214277 12860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:30.215394 12860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:30.215500 12860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:30.215605 12860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:30.215678 12860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:30.226341 12860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43651
I20260927 02:46:30.227636 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:30.227825 12989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43651 every 8 connection(s)
I20260927 02:46:30.231184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 12860
I20260927 02:46:30.231350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:30.231649 392 external_mini_cluster.cc:1428] Setting key 52fa25a15e246c1a46f25a58c02913e4
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:30.242560 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:30.229862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54037 (local address 127.0.98.62:33965)
0927 02:46:30.230029 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:30.230035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:30.230926 (+ 891us) server_negotiation.cc:407] Connection header received
0927 02:46:30.232092 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:30.232096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:30.232177 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:30.232308 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:30.233619 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.234138 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.235524 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.235898 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.236003 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:30.238669 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:30.238674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:30.238677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:30.238706 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:30.240346 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:30.240826 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:30.240829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:30.240831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:30.240918 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:30.241250 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:30.241256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:30.241259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:30.241526 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:30.241638 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:30.242129 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 02:46:30.242292 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:46:30.243252 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:30.230228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33965 (local address 127.0.98.2:54037)
0927 02:46:30.230727 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:30.230789 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:30.231813 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:30.232492 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:30.232501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:30.232834 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:30.233477 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:30.233493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.234273 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.234276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:30.235376 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:30.235390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.236086 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.236092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:30.236319 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:30.236941 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:30.236967 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:30.238497 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:30.240458 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:30.240463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:30.240475 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:30.240725 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:30.241007 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:30.241009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:30.241010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:30.241150 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:30.241634 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:30.241639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:30.241891 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 02:46:30.242051 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 02:46:30.242310 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":105,"threads_started":1}
I20260927 02:46:30.244647 12990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33965
I20260927 02:46:30.244891 12990 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:30.245503 12990 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:46:30.246793 12640 ts_manager.cc:194] Registered new tserver with Master: 0c3f8bea28b34642829d8a474ce8b065 (127.0.98.2:43651)
I20260927 02:46:30.247371 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:46:30.270684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33965
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 02:46:30.427273 12995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:30.427634 12995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:30.427757 12995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:30.433416 12995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:30.433552 12995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:30.433701 12995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:30.441323 12995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:53361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:30.443046 12995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:30.444510 12995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:30.456640 13001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:30.457283 13003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:30.457394 13000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:30.458717 12995 server_base.cc:1034] running on GCE node
I20260927 02:46:30.459193 12995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:30.460064 12995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:30.463583 12995 hybrid_clock.cc:648] HybridClock initialized: now 1790477190463499 us; error 93 us; skew 500 ppm
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:30.467727 12995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:30.468943 12995 webserver.cc:460] Webserver started at http://127.0.98.3:34903/ using document root <none> and password file <none>
I20260927 02:46:30.469704 12995 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:30.469856 12995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:30.470149 12995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:30.501253 12995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "f346efe8bb68444aa31df426aab89503"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-gkrl"
server_key: "a5a492de11995b325c1b69bfbe5f7932"
server_key_iv: "9b95aa8c84ea596943412ca5e7947740"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:30.502099 12995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f346efe8bb68444aa31df426aab89503"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-gkrl"
server_key: "a5a492de11995b325c1b69bfbe5f7932"
server_key_iv: "9b95aa8c84ea596943412ca5e7947740"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:30.507115 12995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 02:46:30.531566 13010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:30.533242 12995 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 02:46:30.533365 12995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "f346efe8bb68444aa31df426aab89503"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-gkrl"
server_key: "a5a492de11995b325c1b69bfbe5f7932"
server_key_iv: "9b95aa8c84ea596943412ca5e7947740"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:30.533486 12995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:30.554617 12995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:30.558095 12995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:30.558303 12995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:30.558935 12995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:30.559921 12995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:30.559981 12995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:30.560034 12995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:30.560062 12995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:30.571120 12995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35093
I20260927 02:46:30.571201 13123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35093 every 8 connection(s)
I20260927 02:46:30.572384 12995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:30.582612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 12995
I20260927 02:46:30.582769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:30.583098 392 external_mini_cluster.cc:1428] Setting key 8f8eb8f43bb371187631439594755318
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:30.588896 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:30.576311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35365 (local address 127.0.98.62:33965)
0927 02:46:30.576424 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:30.576428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:30.576447 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:46:30.577097 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:30.577101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:30.577156 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:30.577252 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:30.578751 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.579580 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.581182 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.581523 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.581650 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:30.584595 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:30.584599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:30.584602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:30.584628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:30.586296 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:30.586961 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:30.586965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:30.586967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:30.587060 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:30.587492 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:30.587496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:30.587498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:30.587738 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:30.587836 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:30.588581 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 02:46:30.588722 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:46:30.589860 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:30.575213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33965 (local address 127.0.98.3:35365)
0927 02:46:30.575752 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:30.575811 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:30.576843 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:30.577453 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:30.577462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:30.577937 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:30.578577 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:30.578595 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.579720 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.579725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:30.581040 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:30.581051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.581689 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.581692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:30.581889 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:30.582597 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:30.582617 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:30.584412 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:30.586440 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:30.586446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:30.586461 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:30.586813 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:30.587166 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:30.587169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:30.587171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:30.587363 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:30.587837 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:30.587845 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:30.588279 (+ 434us) client_negotiation.cc:769] Sending connection context
0927 02:46:30.588485 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 02:46:30.588811 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":138,"threads_started":1}
I20260927 02:46:30.591002 13124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33965
I20260927 02:46:30.591248 13124 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:30.591813 13124 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
I20260927 02:46:30.593031 12640 ts_manager.cc:194] Registered new tserver with Master: f346efe8bb68444aa31df426aab89503 (127.0.98.3:35093)
I20260927 02:46:30.593787 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35365
I20260927 02:46:30.597908 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:30.598717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 12721
I20260927 02:46:30.606555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 12860
I20260927 02:46:30.614070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 12995
I20260927 02:46:30.620613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 12608
I20260927 02:46:30.628486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33965
--webserver_interface=127.0.98.62
--webserver_port=36417
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33965
--ranger_config_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53361/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53361/kms with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:30.739198 13127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:30.739473 13127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:30.739522 13127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:30.743533 13127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:30.743592 13127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:30.743610 13127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:30.743626 13127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:30.743641 13127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:30.749068 13127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53361/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33965
--ranger_config_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:30.750011 13127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:30.750962 13127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:30.759676 13133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:30.759676 13135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:30.759681 13132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:30.759728 13127 server_base.cc:1034] running on GCE node
I20260927 02:46:30.760504 13127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:30.761547 13127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:30.762712 13127 hybrid_clock.cc:648] HybridClock initialized: now 1790477190762697 us; error 31 us; skew 500 ppm
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:30.765506 13127 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:30.766399 13127 webserver.cc:460] Webserver started at http://127.0.98.62:36417/ using document root <none> and password file <none>
I20260927 02:46:30.766947 13127 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:30.767046 13127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:46:30 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:30.815677 13127 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 02:46:30.818404 13143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:30.820010 13127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:30.820142 13127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "8c1bd11fc85b4b62a761008ffa0de6af"
format_stamp: "Formatted at 2026-09-27 02:46:28 on dist-test-slave-gkrl"
server_key: "f9329689752c27dbfdaa46c23881ef71"
server_key_iv: "77f2033545d6f2e3570712053604940b"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:30.820847 13127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:30.847437 13127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:30.850939 13127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:30.851172 13127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:30.859689 13127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33965
I20260927 02:46:30.859720 13195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33965 every 8 connection(s)
I20260927 02:46:30.861001 13127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:30.865204 13196 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:46:30.866128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 13127
I20260927 02:46:30.866614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32935
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33965
--webserver_port=32875
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53361/kms with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:30.867163 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Bootstrap starting.
I20260927 02:46:30.883518 13196 log.cc:824] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:30.890106 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:46:30.890543 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Bootstrap complete.
I20260927 02:46:30.894840 13196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260927 02:46:30.895239 13196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1bd11fc85b4b62a761008ffa0de6af, State: Initialized, Role: FOLLOWER
I20260927 02:46:30.895774 13196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260927 02:46:30.895942 13196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:30.896027 13196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:30.896153 13196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:46:30.897993 13196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260927 02:46:30.898327 13196 leader_election.cc:302] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c1bd11fc85b4b62a761008ffa0de6af; no voters:
I20260927 02:46:30.898687 13196 leader_election.cc:288] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:46:30.898768 13200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:46:30.899060 13200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [term 2 LEADER]: Becoming Leader. State: Replica: 8c1bd11fc85b4b62a761008ffa0de6af, State: Running, Role: LEADER
I20260927 02:46:30.899412 13200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260927 02:46:30.900086 13196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:30.900413 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c1bd11fc85b4b62a761008ffa0de6af. Latest consensus state: current_term: 2 leader_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } } }
I20260927 02:46:30.900594 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:30.900823 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1bd11fc85b4b62a761008ffa0de6af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } } }
I20260927 02:46:30.900978 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:30.901382 13204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:30.904597 13204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:30.905064 13196 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:46:30.907470 13204 catalog_manager.cc:1259] Loaded cluster ID: 030e4dd3d8d74a3fa920b57b01a0e3da
I20260927 02:46:30.907641 13204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:30.910115 13204 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:30.910816 13204 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Loaded TSK: 0
I20260927 02:46:30.912207 13204 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:46:31.025457 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:31.025935 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:31.026053 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:31.031634 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:31.031831 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:31.031985 13198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:31.039619 13198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32875
--tserver_master_addrs=127.0.98.62:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:31.041230 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:31.042579 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:31.054649 13233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:31.054653 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:31.055238 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:31.056449 13198 server_base.cc:1034] running on GCE node
I20260927 02:46:31.056888 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:31.057595 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:31.058774 13198 hybrid_clock.cc:648] HybridClock initialized: now 1790477191058758 us; error 30 us; skew 500 ppm
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:31.062243 13198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:31.063243 13198 webserver.cc:460] Webserver started at http://127.0.98.1:32875/ using document root <none> and password file <none>
I20260927 02:46:31.063920 13198 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:31.064007 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:31.084960 13198 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260927 02:46:31.093667 13244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.095396 13198 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260927 02:46:31.095523 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "ea25ee677d384834b8f02c16eb4b1466"
format_stamp: "Formatted at 2026-09-27 02:46:29 on dist-test-slave-gkrl"
server_key: "663e2148cc730903c1f54ea6e4919946"
server_key_iv: "c3be9e2c297997821dbe1096a2461517"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:31.096261 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:31.126576 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:31.130244 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:31.130534 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:31.131153 13198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:31.132083 13198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:31.132151 13198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.132228 13198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:31.132263 13198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.143769 13198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32935
I20260927 02:46:31.144806 13357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32935 every 8 connection(s)
I20260927 02:46:31.145992 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:31.152390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 13198
I20260927 02:46:31.152880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43651
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33965
--webserver_port=43557
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53361/kms with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:31.167594 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:31.148384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33965 (local address 127.0.98.1:42511)
0927 02:46:31.149769 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:31.149810 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:31.150940 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:31.152831 (+ 1891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:31.152841 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:31.153184 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:31.153851 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:31.153873 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.155677 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.155680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:31.157240 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:31.157251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.158425 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.158429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:31.158607 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:31.159312 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:31.159338 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:31.161022 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:31.164151 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:31.164157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:31.164171 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:31.164501 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:31.164878 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:31.164881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:31.164883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:31.165043 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:31.165774 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:31.165780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:31.166069 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 02:46:31.166248 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 02:46:31.166571 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"thread_start_us":130,"threads_started":1}
I20260927 02:46:31.169478 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:31.148218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42511 (local address 127.0.98.62:33965)
0927 02:46:31.149469 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:31.149496 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:31.150129 (+ 633us) server_negotiation.cc:407] Connection header received
0927 02:46:31.151509 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:31.151525 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:31.151973 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:31.152326 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:31.154053 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.155515 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.157393 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.158072 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.158216 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:31.161232 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:31.161239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:31.161255 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:31.161284 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:31.164020 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:31.164633 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:31.164639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:31.164644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:31.164777 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:31.165142 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:31.165145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:31.165147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:31.165602 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:31.165722 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:31.168341 (+ 2619us) server_negotiation.cc:299] Negotiation successful
0927 02:46:31.168622 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1072,"thread_start_us":117,"threads_started":1}
I20260927 02:46:31.172852 13358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33965
I20260927 02:46:31.173084 13358 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.173627 13358 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
W20260927 02:46:31.175859 13358 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33965 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:46:31.176007 13358 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.176168 13358 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
I20260927 02:46:31.177031 13358 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.177178 13358 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
W20260927 02:46:31.177920 13358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:46:31.338150 13362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:31.338604 13362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:31.338722 13362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:31.344748 13362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:31.344895 13362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:31.345018 13362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:31.350629 13362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43557
--tserver_master_addrs=127.0.98.62:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:31.351699 13362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:31.352852 13362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:31.362455 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:31.362455 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:31.362869 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:31.363198 13362 server_base.cc:1034] running on GCE node
I20260927 02:46:31.363570 13362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:31.364286 13362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:31.365463 13362 hybrid_clock.cc:648] HybridClock initialized: now 1790477191365443 us; error 37 us; skew 500 ppm
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:31.368577 13362 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:31.369550 13362 webserver.cc:460] Webserver started at http://127.0.98.2:43557/ using document root <none> and password file <none>
I20260927 02:46:31.370173 13362 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:31.370294 13362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:31.394493 13362 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 02:46:31.397922 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.399648 13362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 02:46:31.399842 13362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "0c3f8bea28b34642829d8a474ce8b065"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-gkrl"
server_key: "78d00f8b740e46306cd87072ea0339ce"
server_key_iv: "4904bc93fd1d591918c7fa6f9f443613"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:31.400815 13362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:31.435851 13362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:31.439281 13362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:31.439513 13362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:31.440306 13362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:31.441335 13362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:31.441435 13362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.441524 13362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:31.441589 13362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.452960 13362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43651
I20260927 02:46:31.453035 13491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43651 every 8 connection(s)
I20260927 02:46:31.454213 13362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:31.462152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 13362
I20260927 02:46:31.462602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35093
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33965
--webserver_port=34903
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53361/kms with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:31.470649 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:31.456485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55291 (local address 127.0.98.62:33965)
0927 02:46:31.456635 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:31.456640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:31.457763 (+ 1123us) server_negotiation.cc:407] Connection header received
0927 02:46:31.458938 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:31.458942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:31.459001 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:31.459110 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:31.460230 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.460758 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.462168 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.462552 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.462670 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:31.465354 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:31.465357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:31.465359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:31.465385 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:31.467300 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:31.467965 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:31.467968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:31.467970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:31.468063 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:31.469501 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:31.469504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:31.469506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:31.469688 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:31.469811 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:31.470341 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 02:46:31.470473 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:46:31.471535 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:31.456994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33965 (local address 127.0.98.2:55291)
0927 02:46:31.457597 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:31.457653 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:31.458722 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:31.459286 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:31.459294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:31.459548 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:31.460089 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:31.460104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.460885 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.460888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:31.462021 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:31.462032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.462724 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.462731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:31.462974 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:31.463685 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:31.463710 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:31.465162 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:31.467430 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:31.467437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:31.467453 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:31.467802 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:31.468186 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:31.468189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:31.468190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:31.469391 (+ 1201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:31.469808 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:31.469815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:31.470092 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 02:46:31.470289 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 02:46:31.470607 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":189,"threads_started":1}
I20260927 02:46:31.472820 13492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33965
I20260927 02:46:31.473078 13492 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.473563 13492 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
W20260927 02:46:31.475044 13492 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33965 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:46:31.475183 13492 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.475356 13492 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
I20260927 02:46:31.476156 13492 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.476334 13492 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
W20260927 02:46:31.477072 13492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:46:31.591274 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:31.591527 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:31.591568 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:31.595712 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:31.595798 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:31.595916 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:31.601611 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34903
--tserver_master_addrs=127.0.98.62:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:31.602646 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:31.603686 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:31.614708 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:31.616776 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:31.618675 13495 server_base.cc:1034] running on GCE node
W20260927 02:46:31.621577 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:31.622046 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:31.622901 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:31.624089 13495 hybrid_clock.cc:648] HybridClock initialized: now 1790477191624054 us; error 48 us; skew 500 ppm
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:31.627727 13495 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:31.628835 13495 webserver.cc:460] Webserver started at http://127.0.98.3:34903/ using document root <none> and password file <none>
I20260927 02:46:31.629660 13495 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:31.629778 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:31.651499 13495 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:46:31.654567 13512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.656328 13495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 02:46:31.656515 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "f346efe8bb68444aa31df426aab89503"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-gkrl"
server_key: "a5a492de11995b325c1b69bfbe5f7932"
server_key_iv: "9b95aa8c84ea596943412ca5e7947740"
server_key_version: "kuduclusterkey@0"
I20260927 02:46:31.657234 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:31.704571 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:31.708642 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:31.708866 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:31.709430 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:31.710199 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:31.710245 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.710292 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:31.710309 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:31.721693 13495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35093
I20260927 02:46:31.722954 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:31.723532 13625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35093 every 8 connection(s)
I20260927 02:46:31.731127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 13495
Sep 27 02:46:31 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:31.738588 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:31.725754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55675 (local address 127.0.98.62:33965)
0927 02:46:31.725928 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:31.725933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:31.726870 (+ 937us) server_negotiation.cc:407] Connection header received
0927 02:46:31.727854 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:31.727858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:31.727939 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:31.728041 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:31.729148 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.729977 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.731220 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.731572 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.731678 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:31.734409 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:31.734412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:31.734415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:31.734440 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:31.736215 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:31.736820 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:31.736823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:31.736826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:31.736927 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:31.737321 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:31.737324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:31.737326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:31.737526 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:31.737597 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:31.738308 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 02:46:31.738430 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:46:31.740064 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:31.731775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.0.98.62:33965)
0927 02:46:31.732155 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:31.732160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:31.732173 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:46:31.732331 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:31.732334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:31.732382 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:31.732476 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:31.733182 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.733693 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.734593 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.734972 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.735098 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:31.736508 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:31.736512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:31.736514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:31.736538 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:31.737946 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:31.738760 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:31.738763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:31.738765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:31.738862 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:31.739139 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:31.739144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:31.739146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:31.739311 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:31.739518 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:31.739656 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:46:31.739766 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":119,"threads_started":1}
I20260927 02:46:31.740140 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:31.726180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33965 (local address 127.0.98.3:55675)
0927 02:46:31.726692 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:31.726740 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:31.727635 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:31.728191 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:31.728198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:31.728456 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:31.728991 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:31.729005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.730103 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.730105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:31.731082 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:31.731093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:31.731839 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:31.731844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:31.732097 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:31.732745 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:31.732764 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:31.734221 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:31.736343 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:31.736350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:31.736366 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:31.736691 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:31.737040 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:31.737043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:31.737045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:31.737208 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:31.737682 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:31.737688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:31.738016 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 02:46:31.738746 (+ 730us) client_negotiation.cc:240] Negotiation successful
0927 02:46:31.739078 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260927 02:46:31.741356 13626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33965
I20260927 02:46:31.741582 13626 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.742008 13626 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
W20260927 02:46:31.743396 13626 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33965 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:46:31.743490 13626 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.743671 13626 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
I20260927 02:46:31.744493 13626 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:31.744645 13626 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
W20260927 02:46:31.745471 13626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:46:32 dist-test-slave-gkrl krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:32.178123 13358 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:32.178431 13358 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
I20260927 02:46:32.179575 13160 ts_manager.cc:194] Registered new tserver with Master: ea25ee677d384834b8f02c16eb4b1466 (127.0.98.1:32935)
I20260927 02:46:32.181304 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42511
I20260927 02:46:32.477327 13492 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:32.477632 13492 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
I20260927 02:46:32.478590 13160 ts_manager.cc:194] Registered new tserver with Master: 0c3f8bea28b34642829d8a474ce8b065 (127.0.98.2:43651)
I20260927 02:46:32.479465 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55291
Sep 27 02:46:32 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:32.745714 13626 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:32.746033 13626 heartbeater.cc:511] Master 127.0.98.62:33965 requested a full tablet report, sending...
I20260927 02:46:32.746817 13159 ts_manager.cc:194] Registered new tserver with Master: f346efe8bb68444aa31df426aab89503 (127.0.98.3:35093)
I20260927 02:46:32.747517 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:32.747737 13159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55675
Sep 27 02:46:32 dist-test-slave-gkrl krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:46:32.768247 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.750147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.0.98.62:33965)
0927 02:46:32.750782 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:32.750787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:32.750801 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:46:32.750839 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:32.750842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:32.750885 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:32.750970 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:32.751874 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.752698 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.760443 (+ 7745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.760742 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.762344 (+ 1602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.762539 (+ 195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:32.762542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:32.762545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:32.762562 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:32.765868 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:32.766478 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:32.766481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:32.766483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:32.766572 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:32.766898 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:32.766901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:32.766903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:32.767103 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:32.767863 (+ 760us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:32.767965 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:46:32.768081 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":95,"threads_started":1}
I20260927 02:46:32.772853 13159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:32.776577 13159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:32.810385 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.791399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35093 (local address 127.0.0.1:39672)
0927 02:46:32.804220 (+ 12821us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:32.804303 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:32.804451 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:32.805128 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:32.805135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:32.805162 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:32.805319 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.805326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.807210 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.807214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.808599 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.808610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.809838 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.809841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.809955 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.809995 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:46:32.810103 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:46:32.810218 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12578,"thread_start_us":106,"threads_started":1}
I20260927 02:46:32.811012 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.792010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.0.98.3:35093)
0927 02:46:32.793905 (+ 1895us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:32.793911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:32.804617 (+ 10706us) server_negotiation.cc:407] Connection header received
0927 02:46:32.804711 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:32.804716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:32.804955 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:32.805113 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:32.805466 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.806585 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.808737 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.809536 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.809635 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.809778 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:32.810784 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0927 02:46:32.810886 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1804,"thread_start_us":82,"threads_started":1}
I20260927 02:46:32.814294 13547 tablet_service.cc:1467] Processing CreateTablet for tablet bc24825790744883a501883da4ccf334 (DEFAULT_TABLE table=test-table [id=7cff78536b5f4c41beaf08deaf609884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:32.815407 13547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc24825790744883a501883da4ccf334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:32.815626 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.791400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43651 (local address 127.0.0.1:40904)
0927 02:46:32.805629 (+ 14229us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:32.805642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:32.805733 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:32.806464 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:32.806467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:32.806484 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:32.806569 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.806574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.807993 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.807997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.809172 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.809183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.815188 (+ 6005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.815192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.815298 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.815324 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:46:32.815400 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:46:32.815474 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14169,"mutex_wait_us":24,"thread_start_us":107,"threads_started":1}
I20260927 02:46:32.820113 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.791666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.0.98.2:43651)
0927 02:46:32.805522 (+ 13856us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:32.805529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:32.805838 (+ 309us) server_negotiation.cc:407] Connection header received
0927 02:46:32.805926 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:32.805932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:32.806127 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:32.806267 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:32.806718 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.807836 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.814064 (+ 6228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.814888 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.814994 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.815125 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:32.819840 (+ 4715us) server_negotiation.cc:299] Negotiation successful
0927 02:46:32.819970 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13741,"thread_start_us":85,"threads_started":1}
I20260927 02:46:32.821338 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.792930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32935 (local address 127.0.0.1:50936)
0927 02:46:32.809416 (+ 16486us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:32.809430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:32.809517 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:32.810214 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:32.810217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:32.810235 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:32.810327 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.810333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.811639 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.811642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.812781 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.812792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.820932 (+ 8140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.820936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.821028 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.821055 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 02:46:32.821131 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:46:32.821201 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16424,"thread_start_us":92,"threads_started":1}
I20260927 02:46:32.821822 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.793317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.0.98.1:32935)
0927 02:46:32.793734 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:32.793742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:32.809619 (+ 15877us) server_negotiation.cc:407] Connection header received
0927 02:46:32.809704 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:32.809709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:32.809903 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:32.810052 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:32.810464 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.811509 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.812921 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.813760 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.813866 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.814005 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:32.821583 (+ 7578us) server_negotiation.cc:299] Negotiation successful
0927 02:46:32.821689 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":92,"threads_started":1}
I20260927 02:46:32.823016 13418 tablet_service.cc:1467] Processing CreateTablet for tablet bc24825790744883a501883da4ccf334 (DEFAULT_TABLE table=test-table [id=7cff78536b5f4c41beaf08deaf609884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:32.824082 13418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc24825790744883a501883da4ccf334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:32.824323 13292 tablet_service.cc:1467] Processing CreateTablet for tablet bc24825790744883a501883da4ccf334 (DEFAULT_TABLE table=test-table [id=7cff78536b5f4c41beaf08deaf609884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:32.825248 13292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc24825790744883a501883da4ccf334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:32.834159 13662 tablet_bootstrap.cc:492] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466: Bootstrap starting.
I20260927 02:46:32.836776 13663 tablet_bootstrap.cc:492] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Bootstrap starting.
I20260927 02:46:32.839184 13663 tablet_bootstrap.cc:654] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:32.840163 13663 log.cc:824] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:32.842381 13663 tablet_bootstrap.cc:492] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: No bootstrap required, opened a new log
I20260927 02:46:32.842592 13663 ts_tablet_manager.cc:1397] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:46:32.848340 13663 raft_consensus.cc:348] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.848834 13663 raft_consensus.cc:374] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:32.848953 13663 raft_consensus.cc:729] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3f8bea28b34642829d8a474ce8b065, State: Initialized, Role: FOLLOWER
I20260927 02:46:32.849531 13663 consensus_queue.cc:260] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.850703 13663 ts_tablet_manager.cc:1428] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 02:46:32.851137 13492 heartbeater.cc:503] Master 127.0.98.62:33965 was elected leader, sending a full tablet report...
I20260927 02:46:32.861058 13662 tablet_bootstrap.cc:654] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:32.861915 13662 log.cc:824] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:32.862768 13661 tablet_bootstrap.cc:492] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Bootstrap starting.
I20260927 02:46:32.864452 13662 tablet_bootstrap.cc:492] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466: No bootstrap required, opened a new log
I20260927 02:46:32.864691 13662 ts_tablet_manager.cc:1397] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20260927 02:46:32.866613 13661 tablet_bootstrap.cc:654] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:32.868700 13661 log.cc:824] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:32.868994 13662 raft_consensus.cc:348] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.869343 13662 raft_consensus.cc:374] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:32.869452 13662 raft_consensus.cc:729] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea25ee677d384834b8f02c16eb4b1466, State: Initialized, Role: FOLLOWER
I20260927 02:46:32.870018 13662 consensus_queue.cc:260] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.871119 13662 ts_tablet_manager.cc:1428] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:46:32.871513 13358 heartbeater.cc:503] Master 127.0.98.62:33965 was elected leader, sending a full tablet report...
I20260927 02:46:32.880427 13661 tablet_bootstrap.cc:492] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: No bootstrap required, opened a new log
I20260927 02:46:32.880748 13661 ts_tablet_manager.cc:1397] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260927 02:46:32.885515 13661 raft_consensus.cc:348] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.885929 13661 raft_consensus.cc:374] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:32.886037 13661 raft_consensus.cc:729] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f346efe8bb68444aa31df426aab89503, State: Initialized, Role: FOLLOWER
I20260927 02:46:32.886637 13661 consensus_queue.cc:260] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.888129 13661 ts_tablet_manager.cc:1428] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260927 02:46:32.888588 13626 heartbeater.cc:503] Master 127.0.98.62:33965 was elected leader, sending a full tablet report...
W20260927 02:46:32.903525 13359 tablet.cc:2369] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:32.941061 13668 raft_consensus.cc:482] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:32.941298 13668 raft_consensus.cc:504] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.942683 13668 leader_election.cc:288] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f346efe8bb68444aa31df426aab89503 (127.0.98.3:35093), 0c3f8bea28b34642829d8a474ce8b065 (127.0.98.2:43651)
I20260927 02:46:32.949333 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.943612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40805 (local address 127.0.98.3:35093)
0927 02:46:32.943730 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:32.943735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:32.944582 (+ 847us) server_negotiation.cc:407] Connection header received
0927 02:46:32.944774 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:32.944777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:32.944825 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:32.944869 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:32.945278 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.946029 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.947648 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.948406 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.948501 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.948611 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:32.949073 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 02:46:32.949165 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:46:32.949960 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.943099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35093 (local address 127.0.98.1:40805)
0927 02:46:32.944494 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:32.944508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:32.944660 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:32.944999 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:32.945001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:32.945017 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:32.945161 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.945167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.946168 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.946171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.947415 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.947428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.948783 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.948788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.948904 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.948935 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 02:46:32.949725 (+ 790us) client_negotiation.cc:240] Negotiation successful
0927 02:46:32.949803 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"thread_start_us":120,"threads_started":1}
I20260927 02:46:32.950742 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc24825790744883a501883da4ccf334" candidate_uuid: "ea25ee677d384834b8f02c16eb4b1466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f346efe8bb68444aa31df426aab89503" is_pre_election: true
I20260927 02:46:32.951076 13577 raft_consensus.cc:2454] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea25ee677d384834b8f02c16eb4b1466 in term 0.
I20260927 02:46:32.951555 13246 leader_election.cc:302] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea25ee677d384834b8f02c16eb4b1466, f346efe8bb68444aa31df426aab89503; no voters:
I20260927 02:46:32.951860 13668 raft_consensus.cc:2781] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:32.951959 13668 raft_consensus.cc:482] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:32.951999 13668 raft_consensus.cc:3037] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:32.953264 13668 raft_consensus.cc:504] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.953614 13668 leader_election.cc:288] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [CANDIDATE]: Term 1 election: Requested vote from peers f346efe8bb68444aa31df426aab89503 (127.0.98.3:35093), 0c3f8bea28b34642829d8a474ce8b065 (127.0.98.2:43651)
I20260927 02:46:32.954186 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc24825790744883a501883da4ccf334" candidate_uuid: "ea25ee677d384834b8f02c16eb4b1466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f346efe8bb68444aa31df426aab89503"
I20260927 02:46:32.954300 13577 raft_consensus.cc:3037] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:32.955549 13577 raft_consensus.cc:2454] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea25ee677d384834b8f02c16eb4b1466 in term 1.
I20260927 02:46:32.955927 13246 leader_election.cc:302] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea25ee677d384834b8f02c16eb4b1466, f346efe8bb68444aa31df426aab89503; no voters:
W20260927 02:46:32.956727 13493 tablet.cc:2369] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:32.957646 13668 raft_consensus.cc:2781] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:32.957861 13668 raft_consensus.cc:686] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [term 1 LEADER]: Becoming Leader. State: Replica: ea25ee677d384834b8f02c16eb4b1466, State: Running, Role: LEADER
I20260927 02:46:32.958194 13668 consensus_queue.cc:237] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } }
I20260927 02:46:32.961565 13159 catalog_manager.cc:5495] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea25ee677d384834b8f02c16eb4b1466 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ea25ee677d384834b8f02c16eb4b1466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea25ee677d384834b8f02c16eb4b1466" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32935 } health_report { overall_health: HEALTHY } } }
I20260927 02:46:32.966181 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.944305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53245 (local address 127.0.98.2:43651)
0927 02:46:32.944453 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:32.944457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:32.945118 (+ 661us) server_negotiation.cc:407] Connection header received
0927 02:46:32.945319 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:32.945323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:32.945375 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:32.945425 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:32.945938 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.946685 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.964852 (+ 18167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.965510 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.965592 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.965653 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:32.965986 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 02:46:32.966064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:46:32.966712 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.943955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43651 (local address 127.0.98.1:53245)
0927 02:46:32.945011 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:32.945025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:32.945195 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:32.945567 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:32.945570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:32.945587 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:32.945714 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.945720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.946840 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.946844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.964702 (+ 17858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:32.964712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:32.965748 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:32.965752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:32.965852 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:32.965874 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:46:32.966494 (+ 620us) client_negotiation.cc:240] Negotiation successful
0927 02:46:32.966577 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":92,"threads_started":1}
I20260927 02:46:32.967484 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc24825790744883a501883da4ccf334" candidate_uuid: "ea25ee677d384834b8f02c16eb4b1466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3f8bea28b34642829d8a474ce8b065" is_pre_election: true
I20260927 02:46:32.967835 13445 raft_consensus.cc:2454] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea25ee677d384834b8f02c16eb4b1466 in term 0.
I20260927 02:46:32.968214 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc24825790744883a501883da4ccf334" candidate_uuid: "ea25ee677d384834b8f02c16eb4b1466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3f8bea28b34642829d8a474ce8b065"
I20260927 02:46:32.968334 13428 raft_consensus.cc:3037] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:32.969674 13428 raft_consensus.cc:2454] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea25ee677d384834b8f02c16eb4b1466 in term 1.
W20260927 02:46:32.975215 13627 tablet.cc:2369] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:33.008775 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.003402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.0.98.1:32935)
0927 02:46:33.003527 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.003531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.003586 (+ 55us) server_negotiation.cc:407] Connection header received
0927 02:46:33.003813 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.003817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.003934 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.004066 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:33.004357 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.005173 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.006410 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.006743 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.006855 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.007210 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:33.007833 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:33.008498 (+ 665us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.008534 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.008598 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:46:33.016363 13577 raft_consensus.cc:1261] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 1 FOLLOWER]: Refusing update from remote peer ea25ee677d384834b8f02c16eb4b1466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:33.016362 13428 raft_consensus.cc:1261] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 1 FOLLOWER]: Refusing update from remote peer ea25ee677d384834b8f02c16eb4b1466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:33.017114 13668 consensus_queue.cc:1035] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3f8bea28b34642829d8a474ce8b065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:33.017921 13672 consensus_queue.cc:1035] T bc24825790744883a501883da4ccf334 P ea25ee677d384834b8f02c16eb4b1466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f346efe8bb68444aa31df426aab89503" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:33.031226 13675 mvcc.cc:205] Tried to move back new op lower bound from 7333794582586814464 to 7333794582359396352. Current Snapshot: MvccSnapshot[applied={T|T < 7333794582586814464}]
I20260927 02:46:33.033649 13678 mvcc.cc:205] Tried to move back new op lower bound from 7333794582586814464 to 7333794582359396352. Current Snapshot: MvccSnapshot[applied={T|T < 7333794582586814464}]
I20260927 02:46:33.045425 13160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:33.045612 13160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:33.048314 13160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c1bd11fc85b4b62a761008ffa0de6af: Sending DeleteTablet for 3 replicas of tablet bc24825790744883a501883da4ccf334
I20260927 02:46:33.049139 13547 tablet_service.cc:1514] Processing DeleteTablet for tablet bc24825790744883a501883da4ccf334 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39672
I20260927 02:46:33.049240 13418 tablet_service.cc:1514] Processing DeleteTablet for tablet bc24825790744883a501883da4ccf334 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40904
I20260927 02:46:33.049726 13692 tablet_replica.cc:330] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: stopping tablet replica
I20260927 02:46:33.049764 13292 tablet_service.cc:1514] Processing DeleteTablet for tablet bc24825790744883a501883da4ccf334 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50936
I20260927 02:46:33.050016 13692 raft_consensus.cc:2229] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:33.050307 13692 raft_consensus.cc:2258] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:33.050647 13693 tablet_replica.cc:330] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: stopping tablet replica
I20260927 02:46:33.050900 13693 raft_consensus.cc:2229] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:33.051153 13693 raft_consensus.cc:2258] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:33.051194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 13198
I20260927 02:46:33.054865 13693 ts_tablet_manager.cc:1905] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:33.055868 13692 ts_tablet_manager.cc:1905] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:33.057634 13693 ts_tablet_manager.cc:1918] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:33.057838 13693 log.cc:1196] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/bc24825790744883a501883da4ccf334
I20260927 02:46:33.058218 13693 ts_tablet_manager.cc:1939] T bc24825790744883a501883da4ccf334 P 0c3f8bea28b34642829d8a474ce8b065: Deleting consensus metadata
I20260927 02:46:33.059304 13692 ts_tablet_manager.cc:1918] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:33.059468 13692 log.cc:1196] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/bc24825790744883a501883da4ccf334
I20260927 02:46:33.059513 13144 catalog_manager.cc:4841] TS 0c3f8bea28b34642829d8a474ce8b065 (127.0.98.2:43651): tablet bc24825790744883a501883da4ccf334 (table test-table [id=7cff78536b5f4c41beaf08deaf609884]) successfully deleted
I20260927 02:46:33.059887 13692 ts_tablet_manager.cc:1939] T bc24825790744883a501883da4ccf334 P f346efe8bb68444aa31df426aab89503: Deleting consensus metadata
W20260927 02:46:33.060806 13147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32935 (error 108)
I20260927 02:46:33.061014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 13362
I20260927 02:46:33.061079 13145 catalog_manager.cc:4841] TS f346efe8bb68444aa31df426aab89503 (127.0.98.3:35093): tablet bc24825790744883a501883da4ccf334 (table test-table [id=7cff78536b5f4c41beaf08deaf609884]) successfully deleted
I20260927 02:46:33.062747 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.061774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32935 (local address 127.0.0.1:50956)
0927 02:46:33.062621 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.062678 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":770}
W20260927 02:46:33.063577 13147 catalog_manager.cc:4584] TS ea25ee677d384834b8f02c16eb4b1466 (127.0.98.1:32935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc24825790744883a501883da4ccf334: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32935: connect: Connection refused (error 111)
I20260927 02:46:33.068794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 13495
I20260927 02:46:33.074584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 13127
I20260927 02:46:33.081463 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:46:33.426636 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:46:33.426741 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:46:33.771680 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:46:33.771 UTC [12194] LOG: received smart shutdown request
2026-09-27 02:46:33.775 UTC [12194] LOG: background worker "logical replication launcher" (PID 12203) exited with exit code 1
2026-09-27 02:46:33.775 UTC [12198] LOG: shutting down
2026-09-27 02:46:33.952 UTC [12194] LOG: database system is shut down
2026-09-27T02:46:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48568 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:34 dist-test-slave-gkrl krb5kdc[13704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:34 dist-test-slave-gkrl krb5kdc[13704](info): set up 2 sockets
Sep 27 02:46:34 dist-test-slave-gkrl krb5kdc[13704](info): commencing operation
krb5kdc: starting...
W20260927 02:46:35.112439 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:35.235404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35661
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35661,127.0.98.61:41055,127.0.98.60:36949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:35.344009 13717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:35.344265 13717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:35.344311 13717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:35.348315 13717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:35.348369 13717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:35.348388 13717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:35.348405 13717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:35.348462 13717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:35.353746 13717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35661,127.0.98.61:41055,127.0.98.60:36949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:35.354753 13717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:35.355767 13717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:35.364214 13722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:35.364306 13723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:35.364387 13717 server_base.cc:1034] running on GCE node
W20260927 02:46:35.364554 13725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:35.365178 13717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:35.366266 13717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:35.367468 13717 hybrid_clock.cc:648] HybridClock initialized: now 1790477195367438 us; error 46 us; skew 500 ppm
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:35.370182 13717 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:35.371105 13717 webserver.cc:460] Webserver started at http://127.0.98.62:37869/ using document root <none> and password file <none>
I20260927 02:46:35.371677 13717 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:35.371769 13717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:35.372051 13717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:35.374137 13717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "2ae082019f754e6ba8891917605afc9e"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "c62a81224aed7a8ddda7d84a152f8d2b"
server_key_iv: "d7d9564d9d2fd018c5b514fea2dc1f7c"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.374728 13717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ae082019f754e6ba8891917605afc9e"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "c62a81224aed7a8ddda7d84a152f8d2b"
server_key_iv: "d7d9564d9d2fd018c5b514fea2dc1f7c"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.378558 13717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 02:46:35.381116 13732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:35.382428 13717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:35.382586 13717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "2ae082019f754e6ba8891917605afc9e"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "c62a81224aed7a8ddda7d84a152f8d2b"
server_key_iv: "d7d9564d9d2fd018c5b514fea2dc1f7c"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.382704 13717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:35.409857 13717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:35.412879 13717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:35.413097 13717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:35.420188 13717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35661
I20260927 02:46:35.420209 13784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35661 every 8 connection(s)
I20260927 02:46:35.421082 13717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:35.421829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 13717
I20260927 02:46:35.421962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:35.422271 392 external_mini_cluster.cc:1428] Setting key ec00ab0860c750a7f78df2603f05a701
I20260927 02:46:35.424522 13785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:35.428179 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:35.444353 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.430288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.0.98.62:35661)
0927 02:46:35.430732 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.430752 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.430966 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:46:35.432004 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.432009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.432304 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.432510 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.433307 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.434617 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.435660 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.436384 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.436553 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.439192 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.439201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.439217 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.439244 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.440944 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.441449 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.441454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.441455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.441561 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.441960 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.441964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.441965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.442405 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.442507 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.442974 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.443214 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":317,"spinlock_wait_cycles":76544,"thread_start_us":192,"threads_started":1}
I20260927 02:46:35.444353 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.430280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35661 (local address 127.0.0.1:41122)
0927 02:46:35.430813 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.430858 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.431651 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.432504 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.432509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.432925 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.433155 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.433168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.434781 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.434784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.435530 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.435540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.436559 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.436562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.436694 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.437290 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.437306 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.439028 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.441064 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.441072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.441080 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.441325 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.441712 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.441714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.441715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.441825 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.442551 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.442557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.442704 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.442914 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.443328 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":158,"spinlock_wait_cycles":4864,"thread_start_us":143,"threads_started":1}
I20260927 02:46:35.446890 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:35.447685 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.447386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60454)
0927 02:46:35.447546 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.447618 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 02:46:35.448011 13734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
I20260927 02:46:35.448931 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.448716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60462)
0927 02:46:35.448834 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.448878 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 02:46:35.449134 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41055: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:46:35.451242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41055
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35661,127.0.98.61:41055,127.0.98.60:36949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:35.498417 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } attempt: 1
I20260927 02:46:35.499402 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.499110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60468)
0927 02:46:35.499262 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.499325 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 02:46:35.500167 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.499950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60484)
0927 02:46:35.500055 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.500107 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:46:35.500335 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41055: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
W20260927 02:46:35.566337 13792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:35.566572 13792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:35.566612 13792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 02:46:35.568521 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } attempt: 2
I20260927 02:46:35.569388 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.569057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60498)
0927 02:46:35.569232 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.569305 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 02:46:35.570168 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.569948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60500)
0927 02:46:35.570068 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.570114 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 02:46:35.570334 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41055: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41055: connect: Connection refused (error 111)
W20260927 02:46:35.571046 13792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:35.571095 13792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:35.571115 13792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:35.571131 13792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:35.571146 13792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:35.576412 13792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35661,127.0.98.61:41055,127.0.98.60:36949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:35.577428 13792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:35.578344 13792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:35.587433 13798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:35.587419 13800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:35.587401 13797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:35.587778 13792 server_base.cc:1034] running on GCE node
I20260927 02:46:35.588476 13792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:35.589600 13792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:35.590780 13792 hybrid_clock.cc:648] HybridClock initialized: now 1790477195590766 us; error 40 us; skew 500 ppm
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:35.593628 13792 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:46:35.594507 13792 webserver.cc:460] Webserver started at http://127.0.98.61:35375/ using document root <none> and password file <none>
I20260927 02:46:35.595055 13792 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:35.595147 13792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:35.595389 13792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:35.597447 13792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/data/instance:
uuid: "daa8122e33a64926ba69a8a3819bd3ef"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "549f0751c0d811cd418ad01d05806474"
server_key_iv: "f4343d552d5c26f2f47fb5e287a80ab5"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.598007 13792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/wal/instance:
uuid: "daa8122e33a64926ba69a8a3819bd3ef"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "549f0751c0d811cd418ad01d05806474"
server_key_iv: "f4343d552d5c26f2f47fb5e287a80ab5"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.601634 13792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:46:35.604005 13807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:35.605229 13792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:46:35.605360 13792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/wal
uuid: "daa8122e33a64926ba69a8a3819bd3ef"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "549f0751c0d811cd418ad01d05806474"
server_key_iv: "f4343d552d5c26f2f47fb5e287a80ab5"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.605469 13792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:35.618408 13792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:35.621376 13792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:35.621560 13792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:35.628551 13792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41055
I20260927 02:46:35.628562 13859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41055 every 8 connection(s)
I20260927 02:46:35.629434 13792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:46:35.632180 13860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:35.635763 13860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:35.637445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 13792
I20260927 02:46:35.637589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-1/wal/instance
I20260927 02:46:35.637868 392 external_mini_cluster.cc:1428] Setting key 7eb52d7beaf23be76ba0fa372faa4e5e
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 02:46:35.650492 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.637668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.0.98.62:35661)
0927 02:46:35.637806 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.637812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.638906 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 02:46:35.640145 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.640151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.640224 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.640325 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.641418 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.642336 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.643465 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.643866 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.644023 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.646357 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.646360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.646362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.646387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.647955 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.648550 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.648554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.648556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.648653 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.649091 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.649094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.649095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.649315 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.649452 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.650142 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.650304 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:46:35.651232 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.638168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35661 (local address 127.0.0.1:41130)
0927 02:46:35.638721 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.638779 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.639941 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.640476 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.640483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.640799 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.641265 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.641278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.642467 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.642469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.643338 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.643347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.644060 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.644063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.644207 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.644806 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.644828 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.646189 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.648105 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.648112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.648127 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.648444 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.648752 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.648755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.648757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.648950 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.649449 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.649456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.649823 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.650083 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.650401 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":128,"threads_started":1}
I20260927 02:46:35.652755 13860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:35.661535 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } attempt: 3
I20260927 02:46:35.663400 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.653440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.98.61:41055)
0927 02:46:35.653712 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.653733 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.653749 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:46:35.653779 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.653783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.653921 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.654021 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.654564 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.655320 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.656532 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.656881 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.656963 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.659842 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.659850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.659860 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.659922 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.661419 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.661937 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.661943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.661945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.662038 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.662374 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.662377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.662378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.662673 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.662803 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.663124 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.663244 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":111,"threads_started":1}
I20260927 02:46:35.663473 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.653262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60508)
0927 02:46:35.653385 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.653397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.653458 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.654055 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.654061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.654252 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.654418 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.654429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.655439 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.655444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.656401 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.656413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.656998 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.657002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.657078 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.657538 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.657550 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.659714 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.661551 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.661557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.661561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.661829 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.662141 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.662144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.662145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.662239 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.662809 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.662828 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.662929 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.663054 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.663181 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 02:46:35.665798 13860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:35.666594 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.666376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36012)
0927 02:46:35.666479 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.666543 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 02:46:35.667281 13808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:46:35.668151 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.667903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36026)
0927 02:46:35.668022 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.668070 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 02:46:35.668295 13860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36949: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
I20260927 02:46:35.669268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36949
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35661,127.0.98.61:41055,127.0.98.60:36949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:35.671271 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.662045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.0.98.61:41055)
0927 02:46:35.662499 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.662509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.662521 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:46:35.662550 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.662552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.662589 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.662680 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.663608 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.664214 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.665055 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.665361 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.665441 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.667935 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.667940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.667942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.667964 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.669659 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.670063 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.670069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.670074 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.670165 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.670471 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.670476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.670480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.670702 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.670779 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.671074 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.671160 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":240,"threads_started":1}
I20260927 02:46:35.671381 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.662001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60516)
0927 02:46:35.662136 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.662151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.662241 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.662706 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.662708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.662908 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.663472 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.663485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.664333 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.664336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.664933 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.664946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.665488 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.665494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.665610 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.666099 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.666121 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.667739 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.669775 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.669777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.669778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.669965 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.670256 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.670258 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.670259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.670355 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.670797 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.670801 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.670918 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.671034 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.671182 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 02:46:35.672369 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:35.673116 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.672812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36030)
0927 02:46:35.672951 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.673014 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 02:46:35.674011 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.673698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36044)
0927 02:46:35.673867 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.673931 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 02:46:35.674149 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36949: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
I20260927 02:46:35.705348 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } attempt: 1
I20260927 02:46:35.706259 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.705906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36060)
0927 02:46:35.706090 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.706164 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 02:46:35.707023 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.706798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36072)
0927 02:46:35.706919 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.706969 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 02:46:35.707172 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36949: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
I20260927 02:46:35.717599 13860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } attempt: 1
I20260927 02:46:35.718365 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.718085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36086)
0927 02:46:35.718221 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.718284 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 02:46:35.719120 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.718899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36094)
0927 02:46:35.719010 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.719061 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:46:35.719297 13860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36949: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
W20260927 02:46:35.779109 13868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:35.779358 13868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:35.779397 13868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 02:46:35.782349 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } attempt: 2
I20260927 02:46:35.783183 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.782835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36106)
0927 02:46:35.783011 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.783084 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 02:46:35.783855 13868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:35.783955 13868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:35.783998 13868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:35.784031 13868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260927 02:46:35.784001 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.783725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36120)
0927 02:46:35.783826 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.783871 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:46:35.784056 13868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:46:35.784145 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36949: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
I20260927 02:46:35.787451 13860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } attempt: 2
I20260927 02:46:35.788154 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.787925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36124)
0927 02:46:35.788037 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.788089 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:46:35.788851 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.788631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36134)
0927 02:46:35.788735 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.788791 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:46:35.788985 13860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36949: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36949: connect: Connection refused (error 111)
I20260927 02:46:35.790006 13868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35661,127.0.98.61:41055,127.0.98.60:36949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:35.791224 13868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:35.792297 13868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:35.800653 13873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:35.800665 13874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:35.800823 13868 server_base.cc:1034] running on GCE node
W20260927 02:46:35.800670 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:35.801608 13868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:35.802709 13868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:35.803947 13868 hybrid_clock.cc:648] HybridClock initialized: now 1790477195803912 us; error 47 us; skew 500 ppm
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:35.806711 13868 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:46:35.807544 13868 webserver.cc:460] Webserver started at http://127.0.98.60:35537/ using document root <none> and password file <none>
I20260927 02:46:35.808137 13868 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:35.808197 13868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:35.808362 13868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:35.810415 13868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/data/instance:
uuid: "e24c38786ebf4a9faaf70e82d4b41be5"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "a8fc3effbe214e4f3e65ff16f5e08343"
server_key_iv: "d79c65ac2301b60de1f15f058b982535"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.810890 13868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/wal/instance:
uuid: "e24c38786ebf4a9faaf70e82d4b41be5"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "a8fc3effbe214e4f3e65ff16f5e08343"
server_key_iv: "d79c65ac2301b60de1f15f058b982535"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.814437 13868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:46:35.816732 13883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:35.817924 13868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:46:35.818037 13868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/wal
uuid: "e24c38786ebf4a9faaf70e82d4b41be5"
format_stamp: "Formatted at 2026-09-27 02:46:35 on dist-test-slave-gkrl"
server_key: "a8fc3effbe214e4f3e65ff16f5e08343"
server_key_iv: "d79c65ac2301b60de1f15f058b982535"
server_key_version: "encryptionkey@0"
I20260927 02:46:35.818117 13868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:35.838138 13868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:35.841113 13868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:35.841265 13868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:35.848121 13868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36949
I20260927 02:46:35.848243 13935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36949 every 8 connection(s)
I20260927 02:46:35.849071 13868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:46:35.851754 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:35.855336 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:35.855981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 13868
I20260927 02:46:35.856101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477031916752-392-0/minicluster-data/master-2/wal/instance
I20260927 02:46:35.856376 392 external_mini_cluster.cc:1428] Setting key 82d614d5940b6465144fd53cdfcaa969
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:46:35.870433 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.857294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.0.98.62:35661)
0927 02:46:35.857423 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.857429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.858212 (+ 783us) server_negotiation.cc:407] Connection header received
0927 02:46:35.859044 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.859047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.859094 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.859169 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.860535 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.861329 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.862723 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.863061 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.863172 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.866010 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.866014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.866017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.866048 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.867784 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.868667 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.868673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.868676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.868769 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.869219 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.869222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.869224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.869426 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.869500 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.870121 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.870242 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:46:35.871106 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.857567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.0.98.60:36949)
0927 02:46:35.858009 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.858026 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.858064 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:46:35.858817 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.858839 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.859060 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.859241 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.860162 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.861515 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.862718 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.863282 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.863385 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.866491 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.866496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.866525 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.866551 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.868491 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.869008 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.869014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.869017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.869103 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.869656 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.869659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.869660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.869873 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.870176 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.870405 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.870547 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":304,"spinlock_wait_cycles":6400,"thread_start_us":125,"threads_started":1}
I20260927 02:46:35.871105 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.857567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35661 (local address 127.0.0.1:41144)
0927 02:46:35.858057 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.858105 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.858817 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.859735 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.859742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.860224 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.860393 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.860403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.861450 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.861452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.862606 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.862618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.863216 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.863221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.863391 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.864191 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.864210 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.865865 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.867934 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.867941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.867948 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.868532 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.868903 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.868907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.868910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.869102 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.869510 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.869516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.869787 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.869929 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.870219 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":1,"spinlock_wait_cycles":22272,"thread_start_us":167,"threads_started":1}
I20260927 02:46:35.872399 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:35.880442 13860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } attempt: 3
I20260927 02:46:35.881310 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.873180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.0.98.61:41055)
0927 02:46:35.873302 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.873306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.873318 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:46:35.873376 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.873379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.873427 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.873509 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.873938 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.874445 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.875176 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.875412 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.875487 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.877538 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.877542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.877544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.877574 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.879210 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.879693 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.879699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.879701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.879798 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.880156 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.880162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.880165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.880403 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.880508 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.880878 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.881035 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:46:35.882040 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.872876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41055 (local address 127.0.0.1:60530)
0927 02:46:35.872984 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.872998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.873073 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.873545 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.873548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.873724 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.873845 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.873852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.874535 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.874538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.875081 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.875090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.875515 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.875518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.875611 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.876121 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.876136 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.877403 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.879347 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.879351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.879354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.879583 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.879924 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.879930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.879935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.880031 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.880518 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.880521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.880639 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.881699 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.881850 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 02:46:35.882992 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:35.883309 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } attempt: 3
I20260927 02:46:35.884243 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.873425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.0.98.62:35661)
0927 02:46:35.873569 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.873575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.873720 (+ 145us) server_negotiation.cc:407] Connection header received
0927 02:46:35.873806 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.873812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.873854 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.873933 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.875073 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.875680 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.876965 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.877256 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.877365 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.879810 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.879813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.879815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.879837 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.882013 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.882484 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.882487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.882489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.882573 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.882880 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.882883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.882885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.883075 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.883276 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.883934 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.884045 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:46:35.890866 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.880853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36146)
0927 02:46:35.880989 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.881007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.881081 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.881375 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.881378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.881538 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.881606 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.881616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.882491 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.882495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.883299 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.883311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.883939 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.883943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.884030 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.884486 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.884500 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.886032 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.888255 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.888258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.888260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.888494 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.889307 (+ 813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.889312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.889314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.889425 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.890197 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.890200 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.890382 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.890489 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.890648 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 02:46:35.891604 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.881074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.0.98.60:36949)
0927 02:46:35.881193 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.881196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.881206 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:35.881232 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.881235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.881267 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.881359 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.881759 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.882365 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.883436 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.883787 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.886199 (+ 2412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.886360 (+ 161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.886365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.886369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.886391 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.888123 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.888609 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.888612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.888614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.888954 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.889559 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.889565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.889569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.890056 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.891181 (+ 1125us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.891279 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.891388 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":54}
Sep 27 02:46:35 dist-test-slave-gkrl krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:35.895634 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.883491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36148)
0927 02:46:35.884591 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.884607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.884685 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.885036 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.885043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.885241 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.885357 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.885365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.886701 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.886704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.887670 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.887680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.888356 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.888360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.888493 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.888963 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.888983 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.891681 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.894195 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.894198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.894200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.894435 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.894698 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.894701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.894702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.894783 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.895153 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.895156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.895230 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.895323 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.895444 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":34,"spinlock_wait_cycles":768}
I20260927 02:46:35.895983 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.883652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36949 (local address 127.0.0.1:36150)
0927 02:46:35.883867 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:35.883911 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:35.883996 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:35.885151 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:35.885154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:35.885365 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:35.885480 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.885487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.887120 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.887124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.888033 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:35.888045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.888746 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.888750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:35.888848 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.889261 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:35.889277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:35.891181 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:35.894191 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.894195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.894198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.894483 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.894855 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:35.894860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:35.894862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:35.894957 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:35.895421 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:35.895424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:35.895530 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 02:46:35.895618 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 02:46:35.895790 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 02:46:35.896256 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.883723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.0.98.60:36949)
0927 02:46:35.884801 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.884810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.884824 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:46:35.884856 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.884859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.884903 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.885467 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.885618 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.886538 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.887807 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.888248 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.888689 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.891844 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.891847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.891850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.891896 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.894084 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.894544 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.894547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.894548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.894617 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.894871 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.894873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.894875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.895051 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.895940 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.896012 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.896112 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":730,"threads_started":1}
I20260927 02:46:35.896683 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.884207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.0.98.60:36949)
0927 02:46:35.884896 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.884905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.884916 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:46:35.884949 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.884955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.885012 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.886027 (+ 1015us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.886135 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.886989 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.888172 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.888607 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.889065 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.891353 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.891358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.891362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.891387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.894051 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.894620 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.894622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.894625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.894735 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.895085 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.895088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.895091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.895294 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.896290 (+ 996us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.896383 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.896501 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":637,"thread_start_us":140,"threads_started":1}
I20260927 02:46:35.896710 13860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef: Bootstrap starting.
I20260927 02:46:35.896889 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.885626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.0.98.61:41055)
0927 02:46:35.885752 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.885757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.885769 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:46:35.885803 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.885805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.885847 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.885932 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.886747 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.887563 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.888760 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.889119 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.889200 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.892328 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.892332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.892335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.892365 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.894057 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.894614 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.894618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.894619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.894726 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.895080 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.895083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.895085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.895302 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.895484 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.896608 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.896732 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:46:35.899731 13860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:35.899852 13785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e: Bootstrap starting.
I20260927 02:46:35.900717 13860 log.cc:824] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:35.902539 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5: Bootstrap starting.
I20260927 02:46:35.902602 13785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:35.902968 13860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef: No bootstrap required, opened a new log
I20260927 02:46:35.903420 13785 log.cc:824] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:35.904930 13936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:35.905447 13785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e: No bootstrap required, opened a new log
I20260927 02:46:35.905683 13936 log.cc:824] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:35.906955 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:35.907001 13860 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.907210 13860 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:35.907267 13860 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa8122e33a64926ba69a8a3819bd3ef, State: Initialized, Role: FOLLOWER
I20260927 02:46:35.907748 13860 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.908185 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5: No bootstrap required, opened a new log
I20260927 02:46:35.909196 13860 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:35.909363 13785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.909578 13785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:35.909638 13785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae082019f754e6ba8891917605afc9e, State: Initialized, Role: FOLLOWER
I20260927 02:46:35.910408 13785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.911664 13785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:46:35.914585 13738 tablet.cc:2369] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:46:35.915192 13965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:35.915274 13965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:35.915570 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:35.915684 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:35.912899 13792 master_runner.cc:186] Error getting master registration for 127.0.98.62:35661: OK, instance_id { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" instance_seqno: 1790477195418196 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:46:35.916817 13792 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 02:46:35.917132 13977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:35.917210 13977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:35.919591 13936 raft_consensus.cc:348] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.919852 13936 raft_consensus.cc:374] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:35.919966 13936 raft_consensus.cc:729] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24c38786ebf4a9faaf70e82d4b41be5, State: Initialized, Role: FOLLOWER
I20260927 02:46:35.920507 13936 consensus_queue.cc:260] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.922147 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.909929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.0.98.62:35661)
0927 02:46:35.910088 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.910093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.911466 (+ 1373us) server_negotiation.cc:407] Connection header received
0927 02:46:35.911676 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.911680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.911742 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.911802 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.912937 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.913782 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.915158 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.915556 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.915643 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.918019 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.918025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.918028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.918057 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.919850 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.920487 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.920491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.920493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.920581 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.921025 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.921028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.921030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.921249 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.921377 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.921869 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.921992 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 02:46:35.923653 13936 sys_catalog.cc:564] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:35.921675 13717 master_runner.cc:186] Error getting master registration for 127.0.98.60:36949: OK, instance_id { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" instance_seqno: 1790477195846083 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:46:35.925475 13717 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 02:46:35.928716 13990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:35.928792 13990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:35.929384 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:35.929495 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:35.930745 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:35.930893 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:35.932353 13868 master_runner.cc:263] No leader master found from master e24c38786ebf4a9faaf70e82d4b41be5
I20260927 02:46:35.932431 13868 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:46:35.934764 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.910360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.0.98.61:41055)
0927 02:46:35.910477 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.910481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.921593 (+ 11112us) server_negotiation.cc:407] Connection header received
0927 02:46:35.922548 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.922552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.922608 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.922669 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.926098 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.926927 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.928129 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.928517 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.928594 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.930533 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.930539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.930543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.930576 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.932454 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.933255 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.933262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.933266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.933368 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.933700 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.933704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.933706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.933921 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.934045 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.934430 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.934564 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:46:35.936426 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:35.911088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.0.98.60:36949)
0927 02:46:35.911220 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:35.911225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:35.924001 (+ 12776us) server_negotiation.cc:407] Connection header received
0927 02:46:35.924186 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:35.924189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:35.924240 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:35.924294 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:35.925279 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.926146 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.929173 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:35.929571 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:35.929666 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:35.932654 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:35.932658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:35.932661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:35.932685 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:35.934430 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.934942 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.934946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.934948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.935033 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:35.935380 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:35.935384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:35.935386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:35.935611 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:35.935763 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:35.936127 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 02:46:35.936228 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:46:35.956429 13957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:35.956629 13957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.957732 13957 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daa8122e33a64926ba69a8a3819bd3ef (127.0.98.61:41055), e24c38786ebf4a9faaf70e82d4b41be5 (127.0.98.60:36949)
I20260927 02:46:35.958606 13834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ae082019f754e6ba8891917605afc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa8122e33a64926ba69a8a3819bd3ef" is_pre_election: true
I20260927 02:46:35.958935 13834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae082019f754e6ba8891917605afc9e in term 0.
I20260927 02:46:35.959360 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ae082019f754e6ba8891917605afc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" is_pre_election: true
I20260927 02:46:35.959654 13910 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae082019f754e6ba8891917605afc9e in term 0.
I20260927 02:46:35.960198 13733 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae082019f754e6ba8891917605afc9e, e24c38786ebf4a9faaf70e82d4b41be5; no voters:
I20260927 02:46:35.962842 13957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:35.962953 13957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:35.963016 13957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:35.964680 13957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.965380 13834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ae082019f754e6ba8891917605afc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa8122e33a64926ba69a8a3819bd3ef"
I20260927 02:46:35.965513 13834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:35.967418 13834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae082019f754e6ba8891917605afc9e in term 1.
I20260927 02:46:35.967746 13734 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae082019f754e6ba8891917605afc9e, daa8122e33a64926ba69a8a3819bd3ef; no voters:
I20260927 02:46:35.968487 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ae082019f754e6ba8891917605afc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24c38786ebf4a9faaf70e82d4b41be5"
I20260927 02:46:35.968622 13910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:35.969995 13910 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae082019f754e6ba8891917605afc9e in term 1.
I20260927 02:46:35.972646 13957 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [CANDIDATE]: Term 1 election: Requested vote from peers daa8122e33a64926ba69a8a3819bd3ef (127.0.98.61:41055), e24c38786ebf4a9faaf70e82d4b41be5 (127.0.98.60:36949)
I20260927 02:46:35.972863 13957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:35.972971 13957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [term 1 LEADER]: Becoming Leader. State: Replica: 2ae082019f754e6ba8891917605afc9e, State: Running, Role: LEADER
I20260927 02:46:35.973351 13957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } }
I20260927 02:46:35.975260 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae082019f754e6ba8891917605afc9e. Latest consensus state: current_term: 1 leader_uuid: "2ae082019f754e6ba8891917605afc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:35.975378 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:35.975775 13994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:35.976908 13994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:35.986783 13910 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [term 1 FOLLOWER]: Refusing update from remote peer 2ae082019f754e6ba8891917605afc9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:35.987514 13992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:35.991689 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae082019f754e6ba8891917605afc9e. Latest consensus state: current_term: 1 leader_uuid: "2ae082019f754e6ba8891917605afc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:35.991848 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:35.992946 13834 raft_consensus.cc:1261] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [term 1 FOLLOWER]: Refusing update from remote peer 2ae082019f754e6ba8891917605afc9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:35.995205 13957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:35.999348 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae082019f754e6ba8891917605afc9e. Latest consensus state: current_term: 1 leader_uuid: "2ae082019f754e6ba8891917605afc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:35.999469 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.005415 13992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ae082019f754e6ba8891917605afc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:36.005564 13992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:36.006835 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ae082019f754e6ba8891917605afc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:36.006979 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.013773 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ae082019f754e6ba8891917605afc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:36.013900 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:36.014401 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ae082019f754e6ba8891917605afc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae082019f754e6ba8891917605afc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35661 } } peers { permanent_uuid: "daa8122e33a64926ba69a8a3819bd3ef" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41055 } } peers { permanent_uuid: "e24c38786ebf4a9faaf70e82d4b41be5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36949 } } }
I20260927 02:46:36.014518 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.016279 13995 mvcc.cc:205] Tried to move back new op lower bound from 7333794594754125824 to 7333794594710179840. Current Snapshot: MvccSnapshot[applied={T|T < 7333794594754125824}]
I20260927 02:46:36.016443 13998 mvcc.cc:205] Tried to move back new op lower bound from 7333794594754125824 to 7333794594710179840. Current Snapshot: MvccSnapshot[applied={T|T < 7333794594754125824}]
I20260927 02:46:36.017377 13996 mvcc.cc:205] Tried to move back new op lower bound from 7333794594754125824 to 7333794594710179840. Current Snapshot: MvccSnapshot[applied={T|T < 7333794594754125824}]
I20260927 02:46:36.020318 13994 catalog_manager.cc:1347] Generated new cluster ID: 8ca68972ad004b5bb53c6d3e4e455094
I20260927 02:46:36.020385 13994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:36.046006 13994 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:36.047286 13994 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:36.060037 13994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ae082019f754e6ba8891917605afc9e: Generated new TSK 0
I20260927 02:46:36.060801 13994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:36.916574 13965 catalog_manager.cc:1259] Loaded cluster ID: 8ca68972ad004b5bb53c6d3e4e455094
I20260927 02:46:36.916663 13965 catalog_manager.cc:1552] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef: loading cluster ID for follower catalog manager: success
I20260927 02:46:36.918355 13965 catalog_manager.cc:1574] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef: acquiring CA information for follower catalog manager: success
I20260927 02:46:36.919458 13965 catalog_manager.cc:1602] T 00000000000000000000000000000000 P daa8122e33a64926ba69a8a3819bd3ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:46:36.929889 13990 catalog_manager.cc:1259] Loaded cluster ID: 8ca68972ad004b5bb53c6d3e4e455094
I20260927 02:46:36.929955 13990 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5: loading cluster ID for follower catalog manager: success
I20260927 02:46:36.931416 13990 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5: acquiring CA information for follower catalog manager: success
I20260927 02:46:36.932512 13990 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e24c38786ebf4a9faaf70e82d4b41be5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:46:37.129869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 13717
I20260927 02:46:37.135388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 13792
I20260927 02:46:37.141152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 13868
2026-09-27T02:46:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3071 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:37 dist-test-slave-gkrl krb5kdc[14012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:37 dist-test-slave-gkrl krb5kdc[14012](info): set up 2 sockets
Sep 27 02:46:37 dist-test-slave-gkrl krb5kdc[14012](info): commencing operation
krb5kdc: starting...
W20260927 02:46:38.180095 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:38 dist-test-slave-gkrl krb5kdc[14012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:38.314822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:38.422912 14025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:38.423211 14025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:38.423283 14025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:38.427246 14025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:38.427330 14025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:38.427368 14025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:38.427405 14025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:38.427439 14025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:38.432535 14025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:38.433526 14025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:38.434585 14025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:38.443087 14033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:38.443073 14025 server_base.cc:1034] running on GCE node
W20260927 02:46:38.443014 14030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:38.443032 14031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:38.443924 14025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:38.445079 14025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:38.446269 14025 hybrid_clock.cc:648] HybridClock initialized: now 1790477198446238 us; error 44 us; skew 500 ppm
Sep 27 02:46:38 dist-test-slave-gkrl krb5kdc[14012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:38.449455 14025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:38.450413 14025 webserver.cc:460] Webserver started at http://127.0.98.62:40101/ using document root <none> and password file <none>
I20260927 02:46:38.450961 14025 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:38.451028 14025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:38.451280 14025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:38.453356 14025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "32f8512dc7844bdc84dd9c06563875b8"
format_stamp: "Formatted at 2026-09-27 02:46:38 on dist-test-slave-gkrl"
server_key: "31515dbbfaa1f44dfd4f3d0ac3890c88"
server_key_iv: "f5792a8dc161b8d717611e7edca8c76a"
server_key_version: "encryptionkey@0"
I20260927 02:46:38.453930 14025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "32f8512dc7844bdc84dd9c06563875b8"
format_stamp: "Formatted at 2026-09-27 02:46:38 on dist-test-slave-gkrl"
server_key: "31515dbbfaa1f44dfd4f3d0ac3890c88"
server_key_iv: "f5792a8dc161b8d717611e7edca8c76a"
server_key_version: "encryptionkey@0"
I20260927 02:46:38.457707 14025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:46:38.460104 14040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:38.461318 14025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:38.461416 14025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "32f8512dc7844bdc84dd9c06563875b8"
format_stamp: "Formatted at 2026-09-27 02:46:38 on dist-test-slave-gkrl"
server_key: "31515dbbfaa1f44dfd4f3d0ac3890c88"
server_key_iv: "f5792a8dc161b8d717611e7edca8c76a"
server_key_version: "encryptionkey@0"
I20260927 02:46:38.461549 14025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:38.478060 14025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:38.481194 14025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:38.481405 14025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:38.488374 14025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38487
I20260927 02:46:38.488368 14092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38487 every 8 connection(s)
I20260927 02:46:38.489377 14025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:38.490723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 14025
I20260927 02:46:38.490900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:38.491192 392 external_mini_cluster.cc:1428] Setting key 1b7b7791d08bde67d7651720e9a326a2
I20260927 02:46:38.492483 14093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:38.499351 14093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8: Bootstrap starting.
Sep 27 02:46:38 dist-test-slave-gkrl krb5kdc[14012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:38.502013 14093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:38.502977 14093 log.cc:824] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:38.505352 14093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8: No bootstrap required, opened a new log
I20260927 02:46:38.506523 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:38.492737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.0.98.62:38487)
0927 02:46:38.493161 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:38.493179 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:38.493221 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:46:38.494094 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:38.494116 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:38.494516 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:38.494894 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:38.495601 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:38.497107 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:38.497995 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:38.498690 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:38.498855 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:38.501647 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:38.501653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:38.501669 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:38.501701 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:38.503631 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:38.504172 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:38.504177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:38.504181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:38.504292 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:38.504586 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:38.504589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:38.504590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:38.504972 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:38.505109 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:38.505477 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 02:46:38.505810 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260927 02:46:38.509509 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:38.509433 14093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8512dc7844bdc84dd9c06563875b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38487 } }
I20260927 02:46:38.509711 14093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:38.509794 14093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f8512dc7844bdc84dd9c06563875b8, State: Initialized, Role: FOLLOWER
I20260927 02:46:38.510190 14093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8512dc7844bdc84dd9c06563875b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38487 } }
I20260927 02:46:38.510313 14093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:38.510440 14093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:38.510571 14093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:38.511602 14093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8512dc7844bdc84dd9c06563875b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38487 } }
I20260927 02:46:38.511998 14093 leader_election.cc:302] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32f8512dc7844bdc84dd9c06563875b8; no voters:
I20260927 02:46:38.512297 14093 leader_election.cc:288] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:38.512424 14098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:38.512668 14098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [term 1 LEADER]: Becoming Leader. State: Replica: 32f8512dc7844bdc84dd9c06563875b8, State: Running, Role: LEADER
I20260927 02:46:38.513000 14098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8512dc7844bdc84dd9c06563875b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38487 } }
I20260927 02:46:38.513532 14093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:38.514572 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32f8512dc7844bdc84dd9c06563875b8. Latest consensus state: current_term: 1 leader_uuid: "32f8512dc7844bdc84dd9c06563875b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8512dc7844bdc84dd9c06563875b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38487 } } }
I20260927 02:46:38.514698 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:38.515072 14107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:38.515055 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32f8512dc7844bdc84dd9c06563875b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8512dc7844bdc84dd9c06563875b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38487 } } }
I20260927 02:46:38.515203 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:38.518307 14107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:38.523701 14107 catalog_manager.cc:1347] Generated new cluster ID: 4902c9a92369496f89ed6bfccdc06400
I20260927 02:46:38.523764 14107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:38.537177 14107 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:38.538082 14107 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:38.543853 14107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32f8512dc7844bdc84dd9c06563875b8: Generated new TSK 0
I20260927 02:46:38.544503 14107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:38.565202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 14025
2026-09-27T02:46:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1422 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:38 dist-test-slave-gkrl krb5kdc[14120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:38 dist-test-slave-gkrl krb5kdc[14120](info): set up 2 sockets
Sep 27 02:46:38 dist-test-slave-gkrl krb5kdc[14120](info): commencing operation
krb5kdc: starting...
W20260927 02:46:39.592212 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:39 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:39.719347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:39.826880 14133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:39.827167 14133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:39.827246 14133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:39.831182 14133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:39.831269 14133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:39.831308 14133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:39.831362 14133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:39.831403 14133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:39.836473 14133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:39.837453 14133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:39.838428 14133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:39.846792 14139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:39.846792 14138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:39.847183 14133 server_base.cc:1034] running on GCE node
W20260927 02:46:39.846791 14141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:39.847779 14133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:39.848804 14133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:39.849960 14133 hybrid_clock.cc:648] HybridClock initialized: now 1790477199849943 us; error 29 us; skew 500 ppm
Sep 27 02:46:39 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:39.855932 14133 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:39.856784 14133 webserver.cc:460] Webserver started at http://127.0.98.62:34471/ using document root <none> and password file <none>
I20260927 02:46:39.857368 14133 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:39.857463 14133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:39.857689 14133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:39.859638 14133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "e4131405aa66445b979f457b998092f4"
format_stamp: "Formatted at 2026-09-27 02:46:39 on dist-test-slave-gkrl"
server_key: "e8f7aad2d841d6320046dbc6e515a453"
server_key_iv: "549ebe2a4ae95a00d62c06c5d19400a3"
server_key_version: "encryptionkey@0"
I20260927 02:46:39.860249 14133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4131405aa66445b979f457b998092f4"
format_stamp: "Formatted at 2026-09-27 02:46:39 on dist-test-slave-gkrl"
server_key: "e8f7aad2d841d6320046dbc6e515a453"
server_key_iv: "549ebe2a4ae95a00d62c06c5d19400a3"
server_key_version: "encryptionkey@0"
I20260927 02:46:39.863783 14133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:39.866137 14148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:39.867429 14133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:39.867578 14133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "e4131405aa66445b979f457b998092f4"
format_stamp: "Formatted at 2026-09-27 02:46:39 on dist-test-slave-gkrl"
server_key: "e8f7aad2d841d6320046dbc6e515a453"
server_key_iv: "549ebe2a4ae95a00d62c06c5d19400a3"
server_key_version: "encryptionkey@0"
I20260927 02:46:39.867687 14133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:39.879974 14133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:39.883249 14133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:39.883466 14133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:39.890847 14133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39687
I20260927 02:46:39.891273 14200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39687 every 8 connection(s)
I20260927 02:46:39.891906 14133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:39.894893 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:39.895398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 14133
I20260927 02:46:39.895541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:39.895846 392 external_mini_cluster.cc:1428] Setting key c2dd80f8f26bfc182a6cf1eccf3f8e79
I20260927 02:46:39.901610 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Bootstrap starting.
I20260927 02:46:39.904009 14201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Neither blocks nor log segments found. Creating new log.
Sep 27 02:46:39 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:39.904922 14201 log.cc:824] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:39.906880 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: No bootstrap required, opened a new log
I20260927 02:46:39.909997 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:39.897275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.0.98.62:39687)
0927 02:46:39.897689 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:39.897704 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:39.897752 (+ 48us) server_negotiation.cc:407] Connection header received
0927 02:46:39.898401 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:39.898418 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:39.898731 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:39.899082 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:39.899937 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:39.900898 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:39.901766 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:39.902229 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:39.902409 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:39.905230 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:39.905236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:39.905249 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:39.905276 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:39.907098 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:39.907580 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:39.907585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:39.907589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:39.907697 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:39.908006 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:39.908008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:39.908009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:39.908438 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:39.908575 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:39.908945 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 02:46:39.909280 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260927 02:46:39.911005 14201 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:39.911278 14201 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:39.911335 14201 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4131405aa66445b979f457b998092f4, State: Initialized, Role: FOLLOWER
I20260927 02:46:39.911789 14201 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:39.911968 14201 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:39.912045 14201 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:39.912199 14201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:39.913089 14201 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:39.913426 14201 leader_election.cc:302] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4131405aa66445b979f457b998092f4; no voters:
I20260927 02:46:39.913749 14201 leader_election.cc:288] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:39.913817 14206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:39.914076 14206 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 1 LEADER]: Becoming Leader. State: Replica: e4131405aa66445b979f457b998092f4, State: Running, Role: LEADER
I20260927 02:46:39.914415 14206 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:39.915016 14201 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:39.916285 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4131405aa66445b979f457b998092f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } } }
I20260927 02:46:39.916311 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4131405aa66445b979f457b998092f4. Latest consensus state: current_term: 1 leader_uuid: "e4131405aa66445b979f457b998092f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } } }
I20260927 02:46:39.916455 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:39.916471 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:39.917477 14215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:39.920064 14215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:39.926331 14215 catalog_manager.cc:1347] Generated new cluster ID: ac7eb4c2cd6746ec9b41573a46325f85
I20260927 02:46:39.926396 14215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:39.937413 14215 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:39.938340 14215 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:39.948110 14215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Generated new TSK 0
I20260927 02:46:39.949004 14215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:46:39.988255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:40.099090 14227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:40.099328 14227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:40.099371 14227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:40.103505 14227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:40.103555 14227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:40.103641 14227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:40.108872 14227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:40.109874 14227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:40.110863 14227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:40.120697 14235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:40.120666 14232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:40.120666 14233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:40.120959 14227 server_base.cc:1034] running on GCE node
I20260927 02:46:40.121503 14227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:40.122172 14227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:40.123344 14227 hybrid_clock.cc:648] HybridClock initialized: now 1790477200123329 us; error 29 us; skew 500 ppm
Sep 27 02:46:40 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:40.126108 14227 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:40.126964 14227 webserver.cc:460] Webserver started at http://127.0.98.1:37043/ using document root <none> and password file <none>
I20260927 02:46:40.127523 14227 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:40.127583 14227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:40.127786 14227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:40.129767 14227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "a06758cf9a5244aa986cf1523833e557"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "6440053d05de7846b28c7cade8caa64c"
server_key_iv: "68910803415c30a20d095aba45006c76"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.130316 14227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a06758cf9a5244aa986cf1523833e557"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "6440053d05de7846b28c7cade8caa64c"
server_key_iv: "68910803415c30a20d095aba45006c76"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.133926 14227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:46:40.136507 14242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.137765 14227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:40.137885 14227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "a06758cf9a5244aa986cf1523833e557"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "6440053d05de7846b28c7cade8caa64c"
server_key_iv: "68910803415c30a20d095aba45006c76"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.137982 14227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:40.151046 14227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:40.154147 14227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:40.154347 14227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:40.154906 14227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:40.155675 14227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:40.155725 14227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.155791 14227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:40.155823 14227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.164490 14227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44189
I20260927 02:46:40.164518 14355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44189 every 8 connection(s)
I20260927 02:46:40.165372 14227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:40.174585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 14227
I20260927 02:46:40.174734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
Sep 27 02:46:40 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:40.175073 392 external_mini_cluster.cc:1428] Setting key 4e6a2f172ff4526c98a65687c2e08c66
I20260927 02:46:40.179530 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.167333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49125 (local address 127.0.98.62:39687)
0927 02:46:40.167645 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.167649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.168621 (+ 972us) server_negotiation.cc:407] Connection header received
0927 02:46:40.169474 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.169477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.169523 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.169605 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:40.170828 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.171384 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.172610 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.172836 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.172923 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.175516 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:40.175519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:40.175522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:40.175549 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:40.177352 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:40.177851 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:40.177857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:40.177862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:40.177955 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:40.178288 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:40.178294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:40.178298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:40.178558 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:40.178634 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.179310 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.179414 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260927 02:46:40.180477 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.167726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:49125)
0927 02:46:40.168472 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.168514 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:40.169254 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:40.169812 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:40.169828 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:40.170174 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:40.170677 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.170691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.171499 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.171501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.172498 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.172507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.172997 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.173000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.173142 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.173712 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:40.173728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:40.175339 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:40.177473 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:40.177478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:40.177490 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:40.177745 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:40.178048 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:40.178050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:40.178051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:40.178195 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:40.178664 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:40.178672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:40.179021 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 02:46:40.179245 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 02:46:40.179611 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":165,"threads_started":1}
I20260927 02:46:40.181671 14356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260927 02:46:40.181931 14356 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:40.182461 14356 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260927 02:46:40.184005 14165 ts_manager.cc:194] Registered new tserver with Master: a06758cf9a5244aa986cf1523833e557 (127.0.98.1:44189)
I20260927 02:46:40.185173 14165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49125
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:46:40.203009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:40.310772 14361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:40.311022 14361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:40.311064 14361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:40.314987 14361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:40.315074 14361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:40.315156 14361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:40.320425 14361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:40.321365 14361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:40.322274 14361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:40.331575 14367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:40.331575 14366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:40.331701 14361 server_base.cc:1034] running on GCE node
W20260927 02:46:40.331634 14369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:40.332229 14361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:40.332841 14361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:40.334026 14361 hybrid_clock.cc:648] HybridClock initialized: now 1790477200333989 us; error 46 us; skew 500 ppm
Sep 27 02:46:40 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:40.337095 14361 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:40.338052 14361 webserver.cc:460] Webserver started at http://127.0.98.2:44481/ using document root <none> and password file <none>
I20260927 02:46:40.338644 14361 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:40.338712 14361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:40.338966 14361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:40.341091 14361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "311846f171af4754807cb99158716bd3"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "c8c8e2d396301e45c10146b499575dc3"
server_key_iv: "dda6395b27ef9d8ee4a060b20056d367"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.341740 14361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "311846f171af4754807cb99158716bd3"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "c8c8e2d396301e45c10146b499575dc3"
server_key_iv: "dda6395b27ef9d8ee4a060b20056d367"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.345626 14361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 02:46:40.348105 14376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.349465 14361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:46:40.349587 14361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "311846f171af4754807cb99158716bd3"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "c8c8e2d396301e45c10146b499575dc3"
server_key_iv: "dda6395b27ef9d8ee4a060b20056d367"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.349711 14361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:40.363008 14361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:40.366142 14361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:40.366339 14361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:40.366900 14361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:40.367666 14361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:40.367712 14361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.367750 14361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:40.367801 14361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.376837 14361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33899
I20260927 02:46:40.376860 14489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33899 every 8 connection(s)
I20260927 02:46:40.377727 14361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:40.379150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 14361
I20260927 02:46:40.379273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:40.379597 392 external_mini_cluster.cc:1428] Setting key e2e2c8f9bc1a346feb2b6c9eb37d77e9
Sep 27 02:46:40 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:46:40.391795 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.379687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42577 (local address 127.0.98.62:39687)
0927 02:46:40.379807 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.379811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.381187 (+ 1376us) server_negotiation.cc:407] Connection header received
0927 02:46:40.382033 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.382039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.382112 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.382205 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:40.383396 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.383977 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.385157 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.385408 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.385536 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.388015 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:40.388018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:40.388020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:40.388042 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:40.389531 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:40.390181 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:40.390183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:40.390185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:40.390271 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:40.390613 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:40.390615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:40.390616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:40.390804 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:40.390884 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.391504 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.391615 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:46:40.392691 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.380180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:42577)
0927 02:46:40.381006 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.381064 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:40.381835 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:40.382371 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:40.382378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:40.382786 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:40.383256 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.383269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.384096 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.384099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.385031 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.385041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.385572 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.385576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.385754 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.386389 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:40.386404 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:40.387816 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:40.389650 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:40.389657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:40.389671 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:40.389959 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:40.390394 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:40.390396 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:40.390398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:40.390528 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:40.390936 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:40.390941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:40.391275 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 02:46:40.391445 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 02:46:40.391779 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":140,"threads_started":1}
I20260927 02:46:40.394079 14490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260927 02:46:40.394359 14490 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:40.394937 14490 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260927 02:46:40.396099 14165 ts_manager.cc:194] Registered new tserver with Master: 311846f171af4754807cb99158716bd3 (127.0.98.2:33899)
I20260927 02:46:40.396709 14165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:46:40.406939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:40.514129 14495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:40.514369 14495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:40.514410 14495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:40.518723 14495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:40.518780 14495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:40.518836 14495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:40.525302 14495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:40.526624 14495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:40.527998 14495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:40.537827 14500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:40.537827 14501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:40.537845 14503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:40.538041 14495 server_base.cc:1034] running on GCE node
I20260927 02:46:40.538417 14495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:40.539065 14495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:40.540244 14495 hybrid_clock.cc:648] HybridClock initialized: now 1790477200540209 us; error 46 us; skew 500 ppm
Sep 27 02:46:40 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:40.543013 14495 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:40.543942 14495 webserver.cc:460] Webserver started at http://127.0.98.3:41219/ using document root <none> and password file <none>
I20260927 02:46:40.544490 14495 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:40.544582 14495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:40.544814 14495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:40.546742 14495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf140ee16c4d433288d6e2cba4c80e95"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "21833ad80d07b8ee5d4d2908603ff2e2"
server_key_iv: "544030ac1af13407cd6aa1e73650f726"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.547305 14495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf140ee16c4d433288d6e2cba4c80e95"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "21833ad80d07b8ee5d4d2908603ff2e2"
server_key_iv: "544030ac1af13407cd6aa1e73650f726"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.550864 14495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:46:40.553401 14510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.554597 14495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:40.554735 14495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "bf140ee16c4d433288d6e2cba4c80e95"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "21833ad80d07b8ee5d4d2908603ff2e2"
server_key_iv: "544030ac1af13407cd6aa1e73650f726"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.554847 14495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:40.577504 14495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:40.580545 14495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:40.580753 14495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:40.581529 14495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:40.582638 14495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:40.582712 14495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.582828 14495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:40.582893 14495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.591821 14495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41321
I20260927 02:46:40.591845 14623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41321 every 8 connection(s)
I20260927 02:46:40.592733 14495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:46:40 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:40.602762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 14495
I20260927 02:46:40.602865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:40.603233 392 external_mini_cluster.cc:1428] Setting key 0ba910f2272d92c4776703224a15d8c8
I20260927 02:46:40.605875 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.594489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35377 (local address 127.0.98.62:39687)
0927 02:46:40.594630 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.594635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.595649 (+ 1014us) server_negotiation.cc:407] Connection header received
0927 02:46:40.596510 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.596513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.596557 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.596637 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:40.597748 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.598313 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.599379 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.599593 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.599688 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.602299 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:40.602302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:40.602304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:40.602325 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:40.603935 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:40.604474 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:40.604481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:40.604483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:40.604568 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:40.604889 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:40.604892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:40.604894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:40.605092 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:40.605192 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.605670 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.605765 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:46:40.606633 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.594851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:35377)
0927 02:46:40.595509 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.595547 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:40.596331 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:40.596788 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:40.596794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:40.597173 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:40.597622 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.597634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.598421 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.598423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.599269 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.599277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.599762 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.599766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.599979 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.600619 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:40.600635 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:40.602109 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:40.604057 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:40.604065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:40.604083 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:40.604394 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:40.604659 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:40.604662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:40.604664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:40.604797 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:40.605230 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:40.605239 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:40.605460 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 02:46:40.605611 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 02:46:40.605834 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":103,"threads_started":1}
I20260927 02:46:40.607584 14624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260927 02:46:40.607820 14624 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:40.608328 14624 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260927 02:46:40.609244 14165 ts_manager.cc:194] Registered new tserver with Master: bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321)
I20260927 02:46:40.609745 14165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35377
I20260927 02:46:40.616963 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:40.625865 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.618888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.0.98.62:39687)
0927 02:46:40.619016 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.619020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.619285 (+ 265us) server_negotiation.cc:407] Connection header received
0927 02:46:40.619378 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.619380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.619414 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.619477 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:40.620274 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.620743 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.621558 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.621760 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.621833 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.623089 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:40.623091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:40.623093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:40.623110 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:40.624371 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:40.624767 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:40.624770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:40.624771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:40.624844 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:40.625069 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:40.625071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:40.625072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:40.625265 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:40.625363 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.625557 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.625654 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:46:40.628621 14165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:40.631462 14165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:40.645776 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.640715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.0.98.1:44189)
0927 02:46:40.641057 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.641068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.641085 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:46:40.641178 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.641183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.641364 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.641575 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:40.641988 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.643058 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.644570 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.645094 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.645219 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.645363 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.645531 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.645622 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":81,"threads_started":1}
I20260927 02:46:40.645766 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.640515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44189 (local address 127.0.0.1:49232)
0927 02:46:40.640768 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.640781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:40.640884 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:40.641733 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:40.641736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:40.641757 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:40.641855 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.641861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.643188 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.643191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.644450 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.644457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.645217 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.645221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.645328 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.645364 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 02:46:40.645499 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 02:46:40.645622 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":90,"thread_start_us":109,"threads_started":1}
I20260927 02:46:40.646061 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.639545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41321 (local address 127.0.0.1:60612)
0927 02:46:40.640536 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.640591 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:40.640721 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:40.641333 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:40.641339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:40.641414 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:40.641564 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.641570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.643042 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.643045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.644315 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.644326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.645482 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.645486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.645732 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.645785 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 02:46:40.645858 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:46:40.645930 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":80,"thread_start_us":91,"threads_started":1}
I20260927 02:46:40.646166 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.639540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.0.98.2:33899)
0927 02:46:40.640424 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.640436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.640998 (+ 562us) server_negotiation.cc:407] Connection header received
0927 02:46:40.641268 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.641273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.641422 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.641565 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:40.641778 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.642664 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.644302 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.645118 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.645281 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.645416 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.645919 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.646006 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":115,"threads_started":1}
I20260927 02:46:40.646482 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.639742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.0.98.3:41321)
0927 02:46:40.640265 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.640279 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.640855 (+ 576us) server_negotiation.cc:407] Connection header received
0927 02:46:40.640944 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.640952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.641177 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.641349 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:40.641690 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.642876 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.644435 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.645173 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.645292 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.645416 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.646233 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.646332 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":151,"threads_started":1}
I20260927 02:46:40.646756 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.639370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33899 (local address 127.0.0.1:58852)
0927 02:46:40.640915 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.640928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:40.641060 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:40.641566 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:40.641568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:40.641584 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:40.641689 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.641695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.642785 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.642791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.644187 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.644196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.645482 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.645486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.645728 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.645786 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 02:46:40.646542 (+ 756us) client_negotiation.cc:240] Negotiation successful
0927 02:46:40.646631 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":152,"thread_start_us":74,"threads_started":1}
I20260927 02:46:40.649173 14290 tablet_service.cc:1467] Processing CreateTablet for tablet a411c93d82154123992d8c22902ef9ef (DEFAULT_TABLE table=test-table [id=6c6d2bb2d1874e7691ca16ba3f62b56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:40.649174 14558 tablet_service.cc:1467] Processing CreateTablet for tablet a411c93d82154123992d8c22902ef9ef (DEFAULT_TABLE table=test-table [id=6c6d2bb2d1874e7691ca16ba3f62b56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:40.650290 14290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a411c93d82154123992d8c22902ef9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:40.650293 14558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a411c93d82154123992d8c22902ef9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:40.654332 14424 tablet_service.cc:1467] Processing CreateTablet for tablet a411c93d82154123992d8c22902ef9ef (DEFAULT_TABLE table=test-table [id=6c6d2bb2d1874e7691ca16ba3f62b56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:40.655334 14424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a411c93d82154123992d8c22902ef9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:40.657581 14642 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Bootstrap starting.
I20260927 02:46:40.658072 14641 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: Bootstrap starting.
I20260927 02:46:40.660512 14641 tablet_bootstrap.cc:654] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:40.661772 14642 tablet_bootstrap.cc:654] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:40.661859 14641 log.cc:824] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:40.662384 14643 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: Bootstrap starting.
I20260927 02:46:40.663002 14642 log.cc:824] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:40.664122 14641 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: No bootstrap required, opened a new log
I20260927 02:46:40.664352 14641 ts_tablet_manager.cc:1397] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:46:40.664850 14643 tablet_bootstrap.cc:654] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:40.665055 14642 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: No bootstrap required, opened a new log
I20260927 02:46:40.665241 14642 ts_tablet_manager.cc:1397] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 02:46:40.665638 14643 log.cc:824] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:40.667534 14643 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: No bootstrap required, opened a new log
I20260927 02:46:40.668040 14643 ts_tablet_manager.cc:1397] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:46:40.668000 14641 raft_consensus.cc:348] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.668308 14641 raft_consensus.cc:374] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:40.668396 14641 raft_consensus.cc:729] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf140ee16c4d433288d6e2cba4c80e95, State: Initialized, Role: FOLLOWER
I20260927 02:46:40.668849 14641 consensus_queue.cc:260] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.668920 14642 raft_consensus.cc:348] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.669217 14642 raft_consensus.cc:374] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:40.669306 14642 raft_consensus.cc:729] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06758cf9a5244aa986cf1523833e557, State: Initialized, Role: FOLLOWER
I20260927 02:46:40.669485 14624 heartbeater.cc:503] Master 127.0.98.62:39687 was elected leader, sending a full tablet report...
I20260927 02:46:40.669817 14641 ts_tablet_manager.cc:1428] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:46:40.669828 14642 consensus_queue.cc:260] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.670653 14356 heartbeater.cc:503] Master 127.0.98.62:39687 was elected leader, sending a full tablet report...
I20260927 02:46:40.671763 14643 raft_consensus.cc:348] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.672086 14643 raft_consensus.cc:374] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:40.672153 14643 raft_consensus.cc:729] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311846f171af4754807cb99158716bd3, State: Initialized, Role: FOLLOWER
I20260927 02:46:40.672580 14643 consensus_queue.cc:260] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.673179 14490 heartbeater.cc:503] Master 127.0.98.62:39687 was elected leader, sending a full tablet report...
I20260927 02:46:40.673471 14643 ts_tablet_manager.cc:1428] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:46:40.673049 14642 ts_tablet_manager.cc:1428] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260927 02:46:40.762229 14648 raft_consensus.cc:482] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:40.762413 14648 raft_consensus.cc:504] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.763538 14648 leader_election.cc:288] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311846f171af4754807cb99158716bd3 (127.0.98.2:33899), bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321)
I20260927 02:46:40.767480 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.763822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49335 (local address 127.0.98.2:33899)
0927 02:46:40.763981 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.763987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.764110 (+ 123us) server_negotiation.cc:407] Connection header received
0927 02:46:40.764335 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.764341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.764402 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.764470 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:40.764795 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.765318 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.766484 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.766914 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.767029 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.767097 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.767277 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.767355 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:46:40.767489 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.763705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33899 (local address 127.0.98.1:49335)
0927 02:46:40.764028 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.764044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:40.764190 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:40.764511 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:40.764516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:40.764538 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:40.764667 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.764673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.765432 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.765438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.766353 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.766362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.767038 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.767041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.767139 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.767162 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:46:40.767232 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:46:40.767324 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":22,"thread_start_us":106,"threads_started":1}
I20260927 02:46:40.767524 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.764058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35187 (local address 127.0.98.3:41321)
0927 02:46:40.764186 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.764190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.764250 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:46:40.764417 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.764419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.764490 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.764568 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:40.764866 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.765357 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.766514 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.766921 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.767045 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.767108 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.767339 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.767416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:46:40.767500 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.763816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41321 (local address 127.0.98.1:35187)
0927 02:46:40.764175 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.764189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:40.764312 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:40.764609 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:40.764611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:40.764625 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:40.764755 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.764763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.765498 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.765501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.766358 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:40.766366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.767069 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.767073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:40.767184 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.767201 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:46:40.767300 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:46:40.767362 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":6,"thread_start_us":141,"threads_started":1}
I20260927 02:46:40.768149 14578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a411c93d82154123992d8c22902ef9ef" candidate_uuid: "a06758cf9a5244aa986cf1523833e557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf140ee16c4d433288d6e2cba4c80e95" is_pre_election: true
I20260927 02:46:40.768311 14444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a411c93d82154123992d8c22902ef9ef" candidate_uuid: "a06758cf9a5244aa986cf1523833e557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311846f171af4754807cb99158716bd3" is_pre_election: true
I20260927 02:46:40.768488 14578 raft_consensus.cc:2454] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06758cf9a5244aa986cf1523833e557 in term 0.
I20260927 02:46:40.768625 14444 raft_consensus.cc:2454] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06758cf9a5244aa986cf1523833e557 in term 0.
I20260927 02:46:40.768946 14246 leader_election.cc:302] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a06758cf9a5244aa986cf1523833e557, bf140ee16c4d433288d6e2cba4c80e95; no voters:
I20260927 02:46:40.769227 14648 raft_consensus.cc:2781] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:40.769312 14648 raft_consensus.cc:482] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:40.769354 14648 raft_consensus.cc:3037] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:40.770361 14648 raft_consensus.cc:504] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.770740 14648 leader_election.cc:288] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 1 election: Requested vote from peers 311846f171af4754807cb99158716bd3 (127.0.98.2:33899), bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321)
I20260927 02:46:40.771046 14444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a411c93d82154123992d8c22902ef9ef" candidate_uuid: "a06758cf9a5244aa986cf1523833e557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311846f171af4754807cb99158716bd3"
I20260927 02:46:40.771088 14578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a411c93d82154123992d8c22902ef9ef" candidate_uuid: "a06758cf9a5244aa986cf1523833e557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf140ee16c4d433288d6e2cba4c80e95"
I20260927 02:46:40.771168 14444 raft_consensus.cc:3037] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:40.771193 14578 raft_consensus.cc:3037] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:40.772480 14578 raft_consensus.cc:2454] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06758cf9a5244aa986cf1523833e557 in term 1.
I20260927 02:46:40.772480 14444 raft_consensus.cc:2454] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06758cf9a5244aa986cf1523833e557 in term 1.
I20260927 02:46:40.772770 14245 leader_election.cc:302] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311846f171af4754807cb99158716bd3, a06758cf9a5244aa986cf1523833e557; no voters:
I20260927 02:46:40.772990 14648 raft_consensus.cc:2781] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:40.773231 14648 raft_consensus.cc:686] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 1 LEADER]: Becoming Leader. State: Replica: a06758cf9a5244aa986cf1523833e557, State: Running, Role: LEADER
I20260927 02:46:40.773550 14648 consensus_queue.cc:237] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:40.776841 14165 catalog_manager.cc:5495] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 reported cstate change: term changed from 0 to 1, leader changed from <none> to a06758cf9a5244aa986cf1523833e557 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a06758cf9a5244aa986cf1523833e557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } health_report { overall_health: HEALTHY } } }
I20260927 02:46:40.795118 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.791799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.0.98.1:44189)
0927 02:46:40.791954 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:40.791958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:40.791976 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:46:40.792138 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:40.792141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:40.792193 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:40.792285 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:40.792496 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.792975 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.793800 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:40.793994 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:40.794065 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:40.794370 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:40.794755 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:40.794842 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:40.794968 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:46:40.795015 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:46:40.800741 14578 raft_consensus.cc:1261] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 1 FOLLOWER]: Refusing update from remote peer a06758cf9a5244aa986cf1523833e557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:40.800741 14444 raft_consensus.cc:1261] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 1 FOLLOWER]: Refusing update from remote peer a06758cf9a5244aa986cf1523833e557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:40.801465 14652 consensus_queue.cc:1035] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:40.801693 14648 consensus_queue.cc:1035] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:40.809091 14657 mvcc.cc:205] Tried to move back new op lower bound from 7333794614472396800 to 7333794614371151872. Current Snapshot: MvccSnapshot[applied={T|T < 7333794614472396800}]
I20260927 02:46:40.816963 14165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:40.817128 14165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:40.819459 14165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Sending DeleteTablet for 3 replicas of tablet a411c93d82154123992d8c22902ef9ef
I20260927 02:46:40.820255 14424 tablet_service.cc:1514] Processing DeleteTablet for tablet a411c93d82154123992d8c22902ef9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58852
I20260927 02:46:40.820338 14558 tablet_service.cc:1514] Processing DeleteTablet for tablet a411c93d82154123992d8c22902ef9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60612
I20260927 02:46:40.820477 14290 tablet_service.cc:1514] Processing DeleteTablet for tablet a411c93d82154123992d8c22902ef9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49232
I20260927 02:46:40.820881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 14227
I20260927 02:46:40.820936 14667 tablet_replica.cc:330] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: stopping tablet replica
I20260927 02:46:40.821241 14667 raft_consensus.cc:2229] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:46:40.822034 14665 tablet_replica.cc:330] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: stopping tablet replica
I20260927 02:46:40.822304 14665 raft_consensus.cc:2229] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:40.822592 14665 raft_consensus.cc:2258] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:40.822962 14666 tablet_replica.cc:330] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: stopping tablet replica
I20260927 02:46:40.823233 14666 raft_consensus.cc:2229] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:40.823489 14666 raft_consensus.cc:2258] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:40.823977 14665 ts_tablet_manager.cc:1905] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:40.824744 14666 ts_tablet_manager.cc:1905] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:40.826538 14665 ts_tablet_manager.cc:1918] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:40.826628 14665 log.cc:1196] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/a411c93d82154123992d8c22902ef9ef
I20260927 02:46:40.826713 14666 ts_tablet_manager.cc:1918] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:40.826821 14666 log.cc:1196] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/a411c93d82154123992d8c22902ef9ef
I20260927 02:46:40.826997 14665 ts_tablet_manager.cc:1939] T a411c93d82154123992d8c22902ef9ef P 311846f171af4754807cb99158716bd3: Deleting consensus metadata
I20260927 02:46:40.827179 14666 ts_tablet_manager.cc:1939] T a411c93d82154123992d8c22902ef9ef P bf140ee16c4d433288d6e2cba4c80e95: Deleting consensus metadata
I20260927 02:46:40.828253 14152 catalog_manager.cc:4841] TS bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321): tablet a411c93d82154123992d8c22902ef9ef (table test-table [id=6c6d2bb2d1874e7691ca16ba3f62b56f]) successfully deleted
I20260927 02:46:40.828318 14151 catalog_manager.cc:4841] TS 311846f171af4754807cb99158716bd3 (127.0.98.2:33899): tablet a411c93d82154123992d8c22902ef9ef (table test-table [id=6c6d2bb2d1874e7691ca16ba3f62b56f]) successfully deleted
W20260927 02:46:40.828471 14149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44189 (error 108)
I20260927 02:46:40.828688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 14361
I20260927 02:46:40.830711 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:40.830452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44189 (local address 127.0.0.1:49260)
0927 02:46:40.830578 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:40.830646 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 02:46:40.830888 14149 catalog_manager.cc:4584] TS a06758cf9a5244aa986cf1523833e557 (127.0.98.1:44189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a411c93d82154123992d8c22902ef9ef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44189: connect: Connection refused (error 111)
I20260927 02:46:40.834539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 14495
I20260927 02:46:40.839967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 14133
I20260927 02:46:40.846421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39687
--webserver_interface=127.0.98.62
--webserver_port=34471
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:40.954627 14669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:40.954929 14669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:40.955000 14669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:40.958971 14669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:40.959070 14669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:40.959110 14669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:40.959143 14669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:40.959178 14669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:40.964349 14669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:40.965374 14669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:40.966387 14669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:40.979236 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:40.979265 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:40.979255 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:40.979301 14669 server_base.cc:1034] running on GCE node
I20260927 02:46:40.980329 14669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:40.981503 14669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:40.982689 14669 hybrid_clock.cc:648] HybridClock initialized: now 1790477200982652 us; error 52 us; skew 500 ppm
Sep 27 02:46:40 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:40.985709 14669 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:40.986595 14669 webserver.cc:460] Webserver started at http://127.0.98.62:34471/ using document root <none> and password file <none>
I20260927 02:46:40.987147 14669 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:40.987217 14669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:40.990974 14669 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:40.993023 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:40.994383 14669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:40.994540 14669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "e4131405aa66445b979f457b998092f4"
format_stamp: "Formatted at 2026-09-27 02:46:39 on dist-test-slave-gkrl"
server_key: "e8f7aad2d841d6320046dbc6e515a453"
server_key_iv: "549ebe2a4ae95a00d62c06c5d19400a3"
server_key_version: "encryptionkey@0"
I20260927 02:46:40.995138 14669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:41.012985 14669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:41.016263 14669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:41.016464 14669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:41.023826 14669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39687
I20260927 02:46:41.024158 14737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39687 every 8 connection(s)
I20260927 02:46:41.024964 14669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:41.029253 14738 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:46:41.030886 14738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Bootstrap starting.
I20260927 02:46:41.032742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 14669
I20260927 02:46:41.033236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44189
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39687
--webserver_port=37043
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:41.048517 14738 log.cc:824] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:41.056989 14738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:46:41.057402 14738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Bootstrap complete.
I20260927 02:46:41.061470 14738 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:41.061861 14738 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4131405aa66445b979f457b998092f4, State: Initialized, Role: FOLLOWER
I20260927 02:46:41.062342 14738 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:41.062454 14738 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:41.062544 14738 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:41.062659 14738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:46:41.064601 14738 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:41.064937 14738 leader_election.cc:302] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4131405aa66445b979f457b998092f4; no voters:
I20260927 02:46:41.065236 14738 leader_election.cc:288] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:46:41.065371 14742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:46:41.065644 14742 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 2 LEADER]: Becoming Leader. State: Replica: e4131405aa66445b979f457b998092f4, State: Running, Role: LEADER
I20260927 02:46:41.066022 14742 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:41.066509 14738 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:41.067546 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4131405aa66445b979f457b998092f4. Latest consensus state: current_term: 2 leader_uuid: "e4131405aa66445b979f457b998092f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } } }
I20260927 02:46:41.067584 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e4131405aa66445b979f457b998092f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } } }
I20260927 02:46:41.067684 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:41.067685 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:41.071998 14757 catalog_manager.cc:1259] Loaded cluster ID: ac7eb4c2cd6746ec9b41573a46325f85
I20260927 02:46:41.072060 14757 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: loading cluster ID for follower catalog manager: success
W20260927 02:46:41.073009 14757 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260927 02:46:41.073081 14757 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260927 02:46:41.073134 14751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:41.075506 14751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:41.076026 14751 catalog_manager.cc:1259] Loaded cluster ID: ac7eb4c2cd6746ec9b41573a46325f85
I20260927 02:46:41.076088 14751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:41.077143 14751 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 02:46:41.082027 14751 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 02:46:41.087536 14751 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:41.088388 14751 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Loaded TSK: 0
I20260927 02:46:41.089586 14751 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:46:41.146895 14740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:41.147157 14740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:41.147233 14740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:41.151262 14740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:41.151350 14740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:41.151469 14740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:41.156574 14740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37043
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:41.157546 14740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:41.158527 14740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:41.169005 14765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:41.169157 14740 server_base.cc:1034] running on GCE node
W20260927 02:46:41.169034 14766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:41.169034 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:41.170040 14740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:41.170704 14740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:41.171959 14740 hybrid_clock.cc:648] HybridClock initialized: now 1790477201171936 us; error 43 us; skew 500 ppm
Sep 27 02:46:41 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:41.174777 14740 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:41.175801 14740 webserver.cc:460] Webserver started at http://127.0.98.1:37043/ using document root <none> and password file <none>
I20260927 02:46:41.176390 14740 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:41.176463 14740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:41.180411 14740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:46:41.182844 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:41.184362 14740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:41.184504 14740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "a06758cf9a5244aa986cf1523833e557"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "6440053d05de7846b28c7cade8caa64c"
server_key_iv: "68910803415c30a20d095aba45006c76"
server_key_version: "encryptionkey@0"
I20260927 02:46:41.185148 14740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:41.212882 14740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:41.216300 14740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:41.216507 14740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:41.217172 14740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:41.218312 14782 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:46:41.222165 14740 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:46:41.222291 14740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260927 02:46:41.222369 14740 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:46:41.224085 14740 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:46:41.224153 14740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:41.224445 14782 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Bootstrap starting.
I20260927 02:46:41.234985 14740 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44189
I20260927 02:46:41.235080 14889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44189 every 8 connection(s)
I20260927 02:46:41.236071 14740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:41.240154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 14740
I20260927 02:46:41.240592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33899
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39687
--webserver_port=44481
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:41.242914 14782 log.cc:824] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:46:41 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:41.252573 14782 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:46:41.253046 14782 tablet_bootstrap.cc:492] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Bootstrap complete.
I20260927 02:46:41.253050 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.238193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33431 (local address 127.0.98.62:39687)
0927 02:46:41.238575 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.238593 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.239221 (+ 628us) server_negotiation.cc:407] Connection header received
0927 02:46:41.240488 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.240508 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.240937 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.241317 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:41.242624 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.243665 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.244869 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.245337 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.245517 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.247931 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:41.247939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:41.247956 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:41.247982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:41.249832 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.250364 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.250374 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.250378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.250489 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.250836 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.250840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.250841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.251157 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:41.251364 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.251947 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.252208 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
I20260927 02:46:41.253059 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.238058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:33431)
0927 02:46:41.239062 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.239100 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:41.239940 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:41.241405 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:41.241417 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:41.241953 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:41.242452 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.242466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.243825 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.243829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.244742 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.244752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.245483 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.245487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.245668 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.246263 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:41.246278 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:41.247689 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:41.249961 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:41.249966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:41.249978 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:41.250238 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:41.250596 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:41.250598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:41.250600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:41.250734 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:41.251336 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:41.251342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:41.251589 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 02:46:41.251798 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 02:46:41.252167 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":155,"threads_started":1}
I20260927 02:46:41.254594 14782 ts_tablet_manager.cc:1397] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20260927 02:46:41.257490 14890 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260927 02:46:41.257725 14890 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:41.257503 14782 raft_consensus.cc:348] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.258040 14782 raft_consensus.cc:729] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06758cf9a5244aa986cf1523833e557, State: Initialized, Role: FOLLOWER
I20260927 02:46:41.258242 14890 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260927 02:46:41.258571 14782 consensus_queue.cc:260] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.259547 14782 ts_tablet_manager.cc:1428] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:46:41.260949 14702 ts_manager.cc:194] Registered new tserver with Master: a06758cf9a5244aa986cf1523833e557 (127.0.98.1:44189)
I20260927 02:46:41.264504 14702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33431
I20260927 02:46:41.267199 14890 heartbeater.cc:503] Master 127.0.98.62:39687 was elected leader, sending a full tablet report...
Sep 27 02:46:41 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:46:41.275118 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.263535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44189 (local address 127.0.0.1:49264)
0927 02:46:41.263951 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.263997 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:41.264121 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:41.264746 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:41.264753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:41.265176 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:41.265308 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.265315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.266585 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.266589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.267676 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.267688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.269097 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.269114 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.269228 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.269745 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:41.269759 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:41.271371 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:41.273315 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:41.273324 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:41.273329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:41.273610 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:41.273907 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:41.273910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:41.273912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:41.274005 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:41.274491 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:41.274496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:41.274619 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 02:46:41.274764 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 02:46:41.274951 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
I20260927 02:46:41.275260 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.263621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.0.98.1:44189)
0927 02:46:41.264267 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.264277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.264297 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:46:41.264354 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.264359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.264546 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.264688 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:41.265413 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.266468 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.268481 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.269001 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.269084 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.271508 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:41.271516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:41.271526 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:41.271545 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:41.273199 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.273738 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.273743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.273744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.273819 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.274115 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.274118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.274120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.274381 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:41.274481 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.274881 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.275029 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":86,"threads_started":1}
I20260927 02:46:41.276444 14824 tablet_service.cc:1514] Processing DeleteTablet for tablet a411c93d82154123992d8c22902ef9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49264
I20260927 02:46:41.277276 14899 tablet_replica.cc:330] stopping tablet replica
I20260927 02:46:41.277566 14899 raft_consensus.cc:2229] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:41.277809 14899 raft_consensus.cc:2258] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:41.278718 14899 ts_tablet_manager.cc:1905] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:41.282222 14899 ts_tablet_manager.cc:1918] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:41.282320 14899 log.cc:1196] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/a411c93d82154123992d8c22902ef9ef
I20260927 02:46:41.282611 14899 ts_tablet_manager.cc:1939] T a411c93d82154123992d8c22902ef9ef P a06758cf9a5244aa986cf1523833e557: Deleting consensus metadata
I20260927 02:46:41.284396 14686 catalog_manager.cc:4841] TS a06758cf9a5244aa986cf1523833e557 (127.0.98.1:44189): tablet a411c93d82154123992d8c22902ef9ef (table test-table [id=6c6d2bb2d1874e7691ca16ba3f62b56f]) successfully deleted
W20260927 02:46:41.359562 14894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:41.359808 14894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:41.359849 14894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:41.363759 14894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:41.363821 14894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:41.363937 14894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:41.369445 14894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44481
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:41.370528 14894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:41.371562 14894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:41.381527 14905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:41.381529 14904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:41.381529 14907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:41.381621 14894 server_base.cc:1034] running on GCE node
I20260927 02:46:41.382277 14894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:41.382921 14894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:41.384083 14894 hybrid_clock.cc:648] HybridClock initialized: now 1790477201384062 us; error 36 us; skew 500 ppm
Sep 27 02:46:41 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:41.386840 14894 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:41.387832 14894 webserver.cc:460] Webserver started at http://127.0.98.2:44481/ using document root <none> and password file <none>
I20260927 02:46:41.388470 14894 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:41.388584 14894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:41.392380 14894 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:46:41.394428 14914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:41.395641 14894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:41.395780 14894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "311846f171af4754807cb99158716bd3"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "c8c8e2d396301e45c10146b499575dc3"
server_key_iv: "dda6395b27ef9d8ee4a060b20056d367"
server_key_version: "encryptionkey@0"
I20260927 02:46:41.396399 14894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:41.419466 14894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:41.422721 14894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:41.422940 14894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:41.423527 14894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:41.424366 14894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:41.424443 14894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:41.424510 14894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:41.424561 14894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:41.433342 14894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33899
I20260927 02:46:41.433396 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33899 every 8 connection(s)
I20260927 02:46:41.434293 14894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:41.438047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 14894
I20260927 02:46:41.438494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41321
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39687
--webserver_port=41219
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:46:41 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:41.448716 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.436095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39809 (local address 127.0.98.62:39687)
0927 02:46:41.436231 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.436234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.437257 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 02:46:41.438148 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.438151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.438210 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.438323 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:41.439541 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.440443 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.441628 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.442065 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.442204 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.444638 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:41.444644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:41.444648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:41.444685 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:41.446465 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.446987 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.446992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.446995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.447097 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.447434 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.447440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.447444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.447697 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:41.447827 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.448325 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.448475 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:46:41.449286 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.436486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:39809)
0927 02:46:41.437104 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.437142 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:41.437936 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:41.438493 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:41.438500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:41.438915 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:41.439384 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.439397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.440593 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.440597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.441490 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.441499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.442244 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.442247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.442390 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.443001 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:41.443016 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:41.444465 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:41.446594 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:41.446599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:41.446613 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:41.446881 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:41.447198 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:41.447200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:41.447202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:41.447333 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:41.447811 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:41.447816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:41.448082 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 02:46:41.448241 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 02:46:41.448484 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":113,"threads_started":1}
I20260927 02:46:41.450460 15028 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260927 02:46:41.450706 15028 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:41.451171 15028 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260927 02:46:41.452163 14702 ts_manager.cc:194] Registered new tserver with Master: 311846f171af4754807cb99158716bd3 (127.0.98.2:33899)
I20260927 02:46:41.452741 14702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39809
W20260927 02:46:41.548400 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:41.548693 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:41.548771 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:41.552706 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:41.552800 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:41.552886 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:41.557943 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41219
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:41.559007 15031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:41.560041 15031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:41.570122 15037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:41.570389 15031 server_base.cc:1034] running on GCE node
W20260927 02:46:41.570137 15036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:41.570122 15039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:41.570930 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:41.571616 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:41.572801 15031 hybrid_clock.cc:648] HybridClock initialized: now 1790477201572766 us; error 47 us; skew 500 ppm
Sep 27 02:46:41 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:41.575557 15031 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:41.576538 15031 webserver.cc:460] Webserver started at http://127.0.98.3:41219/ using document root <none> and password file <none>
I20260927 02:46:41.577095 15031 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:41.577189 15031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:41.580958 15031 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:46:41.582926 15046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:41.584167 15031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:41.584293 15031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "bf140ee16c4d433288d6e2cba4c80e95"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "21833ad80d07b8ee5d4d2908603ff2e2"
server_key_iv: "544030ac1af13407cd6aa1e73650f726"
server_key_version: "encryptionkey@0"
I20260927 02:46:41.585023 15031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:41.607832 15031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:41.610769 15031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:41.610965 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:41.611544 15031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:41.612404 15031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:41.612476 15031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:41.612542 15031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:41.612591 15031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:41.621290 15031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41321
I20260927 02:46:41.621316 15159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41321 every 8 connection(s)
I20260927 02:46:41.622248 15031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:41.625023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 15031
Sep 27 02:46:41 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:41.634095 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.625679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.0.98.62:39687)
0927 02:46:41.625990 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.625994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.626059 (+ 65us) server_negotiation.cc:407] Connection header received
0927 02:46:41.626200 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.626202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.626243 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.626341 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:41.627314 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.627881 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.628817 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.629037 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.629112 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.630582 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:41.630584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:41.630586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:41.630605 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:41.632243 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.632698 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.632702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.632704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.632805 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.633103 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.633105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.633106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.633275 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:41.633456 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.633645 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.633882 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260927 02:46:41.636608 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.624009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39771 (local address 127.0.98.62:39687)
0927 02:46:41.624141 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.624145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.625317 (+ 1172us) server_negotiation.cc:407] Connection header received
0927 02:46:41.626477 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.626479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.626510 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.626583 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:41.627932 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.628431 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.629955 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.630156 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.630231 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.632876 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:41.632880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:41.632884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:41.632910 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:41.634528 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.635098 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.635101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.635103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.635184 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:41.635517 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:41.635520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:41.635522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:41.635712 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:41.635804 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.636424 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.636508 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:46:41.637419 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.624372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:39771)
0927 02:46:41.625099 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.625158 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:41.626248 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:41.626803 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:41.626814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:41.627319 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:41.627778 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.627790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.628551 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.628554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.629825 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.629837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.630315 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.630319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.630524 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.631094 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:41.631117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:41.632680 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:41.634626 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:41.634631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:41.634645 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:41.634993 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:41.635289 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:41.635291 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:41.635293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:41.635424 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:41.635958 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:41.635965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:41.636190 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 02:46:41.636361 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 02:46:41.636599 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":104,"threads_started":1}
I20260927 02:46:41.638458 15160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260927 02:46:41.638688 15160 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:41.639134 15160 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260927 02:46:41.640110 14702 ts_manager.cc:194] Registered new tserver with Master: bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321)
I20260927 02:46:41.640645 14702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39771
I20260927 02:46:41.647630 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:41.651575 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.648223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.0.98.62:39687)
0927 02:46:41.648371 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.648377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.648534 (+ 157us) server_negotiation.cc:407] Connection header received
0927 02:46:41.648652 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.648654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.648741 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.648819 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:41.649014 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.649516 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.650325 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.650594 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.650671 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.650918 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:41.651250 (+ 332us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:41.651331 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.651389 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.651426 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 02:46:41.652172 14702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:41.655164 14702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:41.662467 14824 tablet_service.cc:1467] Processing CreateTablet for tablet ac188b6bcf984856b2a3685e72db9b55 (DEFAULT_TABLE table=test-table [id=e5e82b46721a42559eccd4478eaeef88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:41.663136 14824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac188b6bcf984856b2a3685e72db9b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:41.665957 14782 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Bootstrap starting.
I20260927 02:46:41.666095 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.661357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41321 (local address 127.0.0.1:54884)
0927 02:46:41.661533 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.661544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:41.661630 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:41.662400 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:41.662403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:41.662431 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:41.662516 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.662521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.663894 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.663900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.664922 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.664930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.665705 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.665708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.665794 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.665832 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:46:41.665899 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:46:41.665976 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 02:46:41.666923 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.661811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33899 (local address 127.0.0.1:52806)
0927 02:46:41.662117 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.662131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:41.662214 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:41.662916 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:41.662918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:41.662937 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:41.663007 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.663012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.664272 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.664277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.665372 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.665382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.666503 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.666507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.666618 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.666642 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:46:41.666698 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:46:41.666762 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260927 02:46:41.666939 14782 tablet_bootstrap.cc:654] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:41.667681 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.662087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.0.98.2:33899)
0927 02:46:41.662457 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.662468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.662492 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:46:41.662567 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.662572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.662753 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.662918 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:41.663102 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.664119 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.665528 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.666335 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.667182 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.667371 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.667424 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.667538 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":101,"threads_started":1}
I20260927 02:46:41.668217 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.661556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.0.98.3:41321)
0927 02:46:41.661894 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.661905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.661927 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:46:41.662001 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.662006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.662228 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.662669 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:41.662831 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.663748 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.665043 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.665580 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.665706 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.667946 (+ 2240us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.667997 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.668090 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":141,"threads_started":1}
I20260927 02:46:41.668592 14782 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: No bootstrap required, opened a new log
I20260927 02:46:41.668676 14782 ts_tablet_manager.cc:1397] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:46:41.669093 14782 raft_consensus.cc:348] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.669215 14782 raft_consensus.cc:374] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:41.669288 14782 raft_consensus.cc:729] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06758cf9a5244aa986cf1523833e557, State: Initialized, Role: FOLLOWER
I20260927 02:46:41.669404 14782 consensus_queue.cc:260] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.669647 14782 ts_tablet_manager.cc:1428] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:41.670783 15094 tablet_service.cc:1467] Processing CreateTablet for tablet ac188b6bcf984856b2a3685e72db9b55 (DEFAULT_TABLE table=test-table [id=e5e82b46721a42559eccd4478eaeef88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:41.670842 14962 tablet_service.cc:1467] Processing CreateTablet for tablet ac188b6bcf984856b2a3685e72db9b55 (DEFAULT_TABLE table=test-table [id=e5e82b46721a42559eccd4478eaeef88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:41.671751 15094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac188b6bcf984856b2a3685e72db9b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:41.671784 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac188b6bcf984856b2a3685e72db9b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:41.677894 15174 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: Bootstrap starting.
I20260927 02:46:41.678757 15173 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: Bootstrap starting.
I20260927 02:46:41.680233 15174 tablet_bootstrap.cc:654] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:41.680871 15173 tablet_bootstrap.cc:654] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:41.681282 15174 log.cc:824] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:41.681578 15173 log.cc:824] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:41.683240 15174 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: No bootstrap required, opened a new log
I20260927 02:46:41.683426 15174 ts_tablet_manager.cc:1397] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:46:41.683491 15173 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: No bootstrap required, opened a new log
I20260927 02:46:41.683660 15173 ts_tablet_manager.cc:1397] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:46:41.686419 15174 raft_consensus.cc:348] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.686419 15173 raft_consensus.cc:348] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.686637 15174 raft_consensus.cc:374] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:41.686668 15173 raft_consensus.cc:374] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:41.686671 15174 raft_consensus.cc:729] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf140ee16c4d433288d6e2cba4c80e95, State: Initialized, Role: FOLLOWER
I20260927 02:46:41.686734 15173 raft_consensus.cc:729] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311846f171af4754807cb99158716bd3, State: Initialized, Role: FOLLOWER
I20260927 02:46:41.687158 15174 consensus_queue.cc:260] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.687208 15173 consensus_queue.cc:260] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.687834 15160 heartbeater.cc:503] Master 127.0.98.62:39687 was elected leader, sending a full tablet report...
I20260927 02:46:41.688176 15174 ts_tablet_manager.cc:1428] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:46:41.688205 15173 ts_tablet_manager.cc:1428] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:46:41.688508 15028 heartbeater.cc:503] Master 127.0.98.62:39687 was elected leader, sending a full tablet report...
W20260927 02:46:41.737354 14891 tablet.cc:2369] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:41.740628 15178 raft_consensus.cc:482] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:41.740803 15178 raft_consensus.cc:504] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.742072 15178 leader_election.cc:288] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321), a06758cf9a5244aa986cf1523833e557 (127.0.98.1:44189)
I20260927 02:46:41.745647 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.742469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42215 (local address 127.0.98.3:41321)
0927 02:46:41.742599 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.742603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.742618 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:46:41.742679 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.742682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.742723 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.742828 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:41.743107 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.743595 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.744694 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.745097 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.745219 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.745279 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.745462 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.745523 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:46:41.745647 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.742265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41321 (local address 127.0.98.2:42215)
0927 02:46:41.742380 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.742397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:41.742484 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:41.742840 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:41.742842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:41.742860 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:41.742978 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.742984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.743709 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.743714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.744568 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.744577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.745217 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.745221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.745332 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.745358 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:46:41.745447 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:46:41.745520 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:46:41.745882 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.742459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46495 (local address 127.0.98.1:44189)
0927 02:46:41.742586 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.742591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.742761 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:46:41.742896 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.742900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.742957 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.743060 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:41.743318 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.743839 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.744905 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.745348 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.745469 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.745631 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.745679 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.745754 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:46:41.746169 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.742366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44189 (local address 127.0.98.2:46495)
0927 02:46:41.742680 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.742690 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:41.742792 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:41.743071 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:41.743074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:41.743088 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:41.743214 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.743219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.744003 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.744006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.744777 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:41.744785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.745470 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.745473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:41.745574 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.745594 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:46:41.745653 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:46:41.746003 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":156,"threads_started":1}
I20260927 02:46:41.746322 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac188b6bcf984856b2a3685e72db9b55" candidate_uuid: "311846f171af4754807cb99158716bd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf140ee16c4d433288d6e2cba4c80e95" is_pre_election: true
I20260927 02:46:41.746649 15114 raft_consensus.cc:2454] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311846f171af4754807cb99158716bd3 in term 0.
I20260927 02:46:41.746763 14844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac188b6bcf984856b2a3685e72db9b55" candidate_uuid: "311846f171af4754807cb99158716bd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06758cf9a5244aa986cf1523833e557" is_pre_election: true
I20260927 02:46:41.747045 14844 raft_consensus.cc:2454] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311846f171af4754807cb99158716bd3 in term 0.
I20260927 02:46:41.747170 14918 leader_election.cc:302] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311846f171af4754807cb99158716bd3, bf140ee16c4d433288d6e2cba4c80e95; no voters:
I20260927 02:46:41.747460 15178 raft_consensus.cc:2781] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:41.747542 15178 raft_consensus.cc:482] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:41.747591 15178 raft_consensus.cc:3037] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:41.748660 15178 raft_consensus.cc:504] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.748956 15178 leader_election.cc:288] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [CANDIDATE]: Term 1 election: Requested vote from peers bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321), a06758cf9a5244aa986cf1523833e557 (127.0.98.1:44189)
I20260927 02:46:41.749240 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac188b6bcf984856b2a3685e72db9b55" candidate_uuid: "311846f171af4754807cb99158716bd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf140ee16c4d433288d6e2cba4c80e95"
I20260927 02:46:41.749285 14844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac188b6bcf984856b2a3685e72db9b55" candidate_uuid: "311846f171af4754807cb99158716bd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06758cf9a5244aa986cf1523833e557"
I20260927 02:46:41.749385 15114 raft_consensus.cc:3037] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:41.749432 14844 raft_consensus.cc:3037] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:41.750341 15114 raft_consensus.cc:2454] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311846f171af4754807cb99158716bd3 in term 1.
I20260927 02:46:41.750494 14844 raft_consensus.cc:2454] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311846f171af4754807cb99158716bd3 in term 1.
I20260927 02:46:41.750615 14918 leader_election.cc:302] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311846f171af4754807cb99158716bd3, bf140ee16c4d433288d6e2cba4c80e95; no voters:
I20260927 02:46:41.750785 15178 raft_consensus.cc:2781] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:41.750998 15178 raft_consensus.cc:686] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 1 LEADER]: Becoming Leader. State: Replica: 311846f171af4754807cb99158716bd3, State: Running, Role: LEADER
I20260927 02:46:41.751308 15178 consensus_queue.cc:237] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:41.754365 14702 catalog_manager.cc:5495] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 311846f171af4754807cb99158716bd3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "311846f171af4754807cb99158716bd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } health_report { overall_health: UNKNOWN } } }
I20260927 02:46:41.774550 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.771394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.0.98.2:33899)
0927 02:46:41.771519 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:41.771523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:41.771537 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:46:41.771637 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:41.771639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:41.771690 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:41.771787 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:41.772024 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.772510 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.773317 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:41.773518 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:41.773593 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:41.773846 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:41.774234 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:41.774324 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:41.774379 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 02:46:41.774426 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:46:41.780507 14844 raft_consensus.cc:1261] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Refusing update from remote peer 311846f171af4754807cb99158716bd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:41.780510 15114 raft_consensus.cc:1261] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 1 FOLLOWER]: Refusing update from remote peer 311846f171af4754807cb99158716bd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:41.781275 15180 consensus_queue.cc:1035] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:41.781457 15178 consensus_queue.cc:1035] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:41.786556 15183 mvcc.cc:205] Tried to move back new op lower bound from 7333794618485260288 to 7333794618375991296. Current Snapshot: MvccSnapshot[applied={T|T < 7333794618485260288}]
I20260927 02:46:41.787760 15184 mvcc.cc:205] Tried to move back new op lower bound from 7333794618485260288 to 7333794618375991296. Current Snapshot: MvccSnapshot[applied={T|T < 7333794618485260288}]
I20260927 02:46:41.796653 14702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:41.796845 14702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:41.799182 14702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Sending DeleteTablet for 3 replicas of tablet ac188b6bcf984856b2a3685e72db9b55
I20260927 02:46:41.799894 15094 tablet_service.cc:1514] Processing DeleteTablet for tablet ac188b6bcf984856b2a3685e72db9b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54884
I20260927 02:46:41.799969 14824 tablet_service.cc:1514] Processing DeleteTablet for tablet ac188b6bcf984856b2a3685e72db9b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49264
I20260927 02:46:41.800324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 14740
I20260927 02:46:41.800335 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet ac188b6bcf984856b2a3685e72db9b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52806
I20260927 02:46:41.800976 15194 tablet_replica.cc:330] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: stopping tablet replica
I20260927 02:46:41.801280 15194 raft_consensus.cc:2229] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:46:41.801666 15194 raft_consensus.cc:2258] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:41.803222 15194 ts_tablet_manager.cc:1905] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:41.804091 15193 tablet_replica.cc:330] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: stopping tablet replica
I20260927 02:46:41.804477 15193 raft_consensus.cc:2229] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:41.804845 15193 raft_consensus.cc:2258] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:41.806167 15194 ts_tablet_manager.cc:1918] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:41.806265 15194 log.cc:1196] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/ac188b6bcf984856b2a3685e72db9b55
I20260927 02:46:41.806496 15193 ts_tablet_manager.cc:1905] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:41.806617 15194 ts_tablet_manager.cc:1939] T ac188b6bcf984856b2a3685e72db9b55 P 311846f171af4754807cb99158716bd3: Deleting consensus metadata
I20260927 02:46:41.807755 14688 catalog_manager.cc:4841] TS 311846f171af4754807cb99158716bd3 (127.0.98.2:33899): tablet ac188b6bcf984856b2a3685e72db9b55 (table test-table [id=e5e82b46721a42559eccd4478eaeef88]) successfully deleted
W20260927 02:46:41.808401 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44189 (error 108)
I20260927 02:46:41.808964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 14894
I20260927 02:46:41.810271 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:41.809982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44189 (local address 127.0.0.1:57842)
0927 02:46:41.810131 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:41.810190 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 02:46:41.810591 15193 ts_tablet_manager.cc:1918] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:41.810745 15193 log.cc:1196] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/ac188b6bcf984856b2a3685e72db9b55
W20260927 02:46:41.810420 14686 catalog_manager.cc:4584] TS a06758cf9a5244aa986cf1523833e557 (127.0.98.1:44189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac188b6bcf984856b2a3685e72db9b55: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44189: connect: Connection refused (error 111)
I20260927 02:46:41.811179 15193 ts_tablet_manager.cc:1939] T ac188b6bcf984856b2a3685e72db9b55 P bf140ee16c4d433288d6e2cba4c80e95: Deleting consensus metadata
I20260927 02:46:41.812281 14689 catalog_manager.cc:4841] TS bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321): tablet ac188b6bcf984856b2a3685e72db9b55 (table test-table [id=e5e82b46721a42559eccd4478eaeef88]) successfully deleted
I20260927 02:46:41.815392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 15031
I20260927 02:46:41.820803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 14669
I20260927 02:46:41.826738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39687
--webserver_interface=127.0.98.62
--webserver_port=34471
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:41.935388 15196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:41.935678 15196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:41.935730 15196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:41.939811 15196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:41.939899 15196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:41.939921 15196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:41.939939 15196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:41.939953 15196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:41.945379 15196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:41.946460 15196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:41.947500 15196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:46:41.956089 15196 server_base.cc:1034] running on GCE node
W20260927 02:46:41.956169 15204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:41.956117 15202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:41.956127 15201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:41.956954 15196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:41.958009 15196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:41.959214 15196 hybrid_clock.cc:648] HybridClock initialized: now 1790477201959200 us; error 28 us; skew 500 ppm
Sep 27 02:46:41 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:41.961923 15196 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:41.962759 15196 webserver.cc:460] Webserver started at http://127.0.98.62:34471/ using document root <none> and password file <none>
I20260927 02:46:41.963297 15196 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:41.963392 15196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:41.967176 15196 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:41.969477 15211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:41.970839 15196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:46:41.970937 15196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "e4131405aa66445b979f457b998092f4"
format_stamp: "Formatted at 2026-09-27 02:46:39 on dist-test-slave-gkrl"
server_key: "e8f7aad2d841d6320046dbc6e515a453"
server_key_iv: "549ebe2a4ae95a00d62c06c5d19400a3"
server_key_version: "encryptionkey@0"
I20260927 02:46:41.971472 15196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:41.989673 15196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:41.992905 15196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:41.993068 15196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:42.000286 15196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39687
I20260927 02:46:42.000613 15263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39687 every 8 connection(s)
I20260927 02:46:42.001637 15196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:42.002655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 15196
I20260927 02:46:42.003239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44189
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39687
--webserver_port=37043
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:42.007596 15264 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:46:42.009655 15264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Bootstrap starting.
I20260927 02:46:42.027452 15264 log.cc:824] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:42.038586 15264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:46:42.039045 15264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4: Bootstrap complete.
I20260927 02:46:42.043195 15264 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:42.043617 15264 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4131405aa66445b979f457b998092f4, State: Initialized, Role: FOLLOWER
I20260927 02:46:42.044176 15264 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:42.044294 15264 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:42.044389 15264 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:42.044523 15264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:46:42.046505 15264 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:42.046859 15264 leader_election.cc:302] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4131405aa66445b979f457b998092f4; no voters:
I20260927 02:46:42.047207 15264 leader_election.cc:288] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:46:42.047278 15268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:46:42.047479 15268 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [term 3 LEADER]: Becoming Leader. State: Replica: e4131405aa66445b979f457b998092f4, State: Running, Role: LEADER
I20260927 02:46:42.047760 15268 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260927 02:46:42.048491 15264 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:42.048666 15269 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4131405aa66445b979f457b998092f4. Latest consensus state: current_term: 3 leader_uuid: "e4131405aa66445b979f457b998092f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } } }
I20260927 02:46:42.048768 15269 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:42.048708 15270 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e4131405aa66445b979f457b998092f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4131405aa66445b979f457b998092f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } } }
I20260927 02:46:42.048837 15270 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4131405aa66445b979f457b998092f4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:42.049134 15272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:42.054314 15272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:42.055001 15272 catalog_manager.cc:1259] Loaded cluster ID: ac7eb4c2cd6746ec9b41573a46325f85
I20260927 02:46:42.055081 15272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 02:46:42.055860 15272 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6b57e0c9ad google::LogMessage::Fail() at ??:0
@ 0x7f6b57e0f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6b57e0c4de google::LogMessage::Flush() at ??:0
@ 0x7f6b57e0da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6b6274bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6b6274cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6b62749579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6b6278d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6b5a05e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6b5a05ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6b5a0603f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6b5a04f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6b5b6936db start_thread at ??:0
@ 0x7f6b5690a61f clone at ??:0
W20260927 02:46:42.145291 15266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:42.145673 15266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:42.145761 15266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:42.151952 15266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:42.152024 15266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:42.152117 15266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:42.159902 15266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37043
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:42.161257 15266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:42.162612 15266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:42.174042 15308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:42.174094 15306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:42.174064 15305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:42.174449 15266 server_base.cc:1034] running on GCE node
I20260927 02:46:42.174993 15266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:42.175810 15266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:42.177021 15266 hybrid_clock.cc:648] HybridClock initialized: now 1790477202176977 us; error 55 us; skew 500 ppm
Sep 27 02:46:42 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:42.180028 15266 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:42.181152 15266 webserver.cc:460] Webserver started at http://127.0.98.1:37043/ using document root <none> and password file <none>
I20260927 02:46:42.181877 15266 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:42.181972 15266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:42.187130 15266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:46:42.189752 15315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:42.191128 15266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:46:42.191291 15266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "a06758cf9a5244aa986cf1523833e557"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "6440053d05de7846b28c7cade8caa64c"
server_key_iv: "68910803415c30a20d095aba45006c76"
server_key_version: "encryptionkey@0"
I20260927 02:46:42.192028 15266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:42.218094 15266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:42.221944 15266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:42.222152 15266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:42.222882 15266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:42.224263 15322 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:46:42.228462 15266 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:46:42.228540 15266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 02:46:42.228603 15266 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:46:42.230753 15266 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:46:42.230813 15266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:46:42.231168 15322 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Bootstrap starting.
I20260927 02:46:42.242152 15266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44189
I20260927 02:46:42.242183 15429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44189 every 8 connection(s)
I20260927 02:46:42.243429 15266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:42.250753 15322 log.cc:824] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:42.251567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 15266
I20260927 02:46:42.252066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33899
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39687
--webserver_port=44481
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:42.259793 15322 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:46:42.260284 15322 tablet_bootstrap.cc:492] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Bootstrap complete.
I20260927 02:46:42.261745 15322 ts_tablet_manager.cc:1397] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260927 02:46:42.264998 15322 raft_consensus.cc:348] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:42.265604 15322 raft_consensus.cc:729] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06758cf9a5244aa986cf1523833e557, State: Initialized, Role: FOLLOWER
I20260927 02:46:42.266139 15322 consensus_queue.cc:260] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:42.267289 15322 ts_tablet_manager.cc:1428] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260927 02:46:42.307797 15432 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:42.245987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:38533)
0927 02:46:42.246676 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.246737 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:42.248023 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:42.306817 (+ 58794us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":210,"threads_started":1}
W20260927 02:46:42.308372 15430 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39687: Client connection negotiation failed: client connection to 127.0.98.62:39687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:46:42.309099 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:42.308827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:33205)
0927 02:46:42.308963 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.309036 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 02:46:42.309892 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:42.309630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:36061)
0927 02:46:42.309748 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.309802 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 02:46:42.310091 15430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:46:42.366642 15434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:42.366955 15434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:42.367031 15434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:42.371323 15434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:42.371414 15434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:42.371501 15434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:42.377131 15434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44481
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:42.378185 15434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:42.379201 15434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:42.389138 15441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:42.389223 15434 server_base.cc:1034] running on GCE node
W20260927 02:46:42.389138 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:42.389132 15443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:42.389806 15434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:42.390491 15434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:42.391692 15434 hybrid_clock.cc:648] HybridClock initialized: now 1790477202391658 us; error 44 us; skew 500 ppm
Sep 27 02:46:42 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:42.394462 15434 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:42.395260 15434 webserver.cc:460] Webserver started at http://127.0.98.2:44481/ using document root <none> and password file <none>
I20260927 02:46:42.395771 15434 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:42.395828 15434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:42.399953 15434 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:46:42.402015 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:42.403400 15434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:42.403496 15434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "311846f171af4754807cb99158716bd3"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "c8c8e2d396301e45c10146b499575dc3"
server_key_iv: "dda6395b27ef9d8ee4a060b20056d367"
server_key_version: "encryptionkey@0"
I20260927 02:46:42.404073 15434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:42.424559 15434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:42.427495 15434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:42.427651 15434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:42.428311 15434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:42.429056 15434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:42.429104 15434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:42.429150 15434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:42.429167 15434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:42.438205 15434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33899
I20260927 02:46:42.438337 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33899 every 8 connection(s)
I20260927 02:46:42.439208 15434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:42.442819 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:42.441345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:38639)
0927 02:46:42.441985 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.442087 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":100,"threads_started":1}
W20260927 02:46:42.443289 15564 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39687: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
I20260927 02:46:42.443819 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:42.443625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:43447)
0927 02:46:42.443726 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.443769 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:46:42.444468 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:42.444276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:58113)
0927 02:46:42.444377 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.444426 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 02:46:42.444651 15564 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:46:42.448732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 15434
I20260927 02:46:42.449134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41321
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39687
--webserver_port=41219
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:42.558730 15567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:42.559018 15567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:42.559093 15567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:42.563138 15567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:42.563230 15567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:42.563318 15567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:42.568660 15567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41219
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:42.569734 15567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:42.570788 15567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:46:42.580821 15567 server_base.cc:1034] running on GCE node
W20260927 02:46:42.580801 15573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:42.580801 15575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:42.580994 15572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:42.581391 15567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:42.582075 15567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:42.583266 15567 hybrid_clock.cc:648] HybridClock initialized: now 1790477202583251 us; error 28 us; skew 500 ppm
Sep 27 02:46:42 dist-test-slave-gkrl krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:42.586103 15567 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:42.587030 15567 webserver.cc:460] Webserver started at http://127.0.98.3:41219/ using document root <none> and password file <none>
I20260927 02:46:42.587572 15567 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:42.587633 15567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:42.592550 15567 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:42.594679 15582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:42.596210 15567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:42.596341 15567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "bf140ee16c4d433288d6e2cba4c80e95"
format_stamp: "Formatted at 2026-09-27 02:46:40 on dist-test-slave-gkrl"
server_key: "21833ad80d07b8ee5d4d2908603ff2e2"
server_key_iv: "544030ac1af13407cd6aa1e73650f726"
server_key_version: "encryptionkey@0"
I20260927 02:46:42.596935 15567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:42.608395 15567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:42.611511 15567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:42.611707 15567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:42.612325 15567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:42.613111 15567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:42.613193 15567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:42.613257 15567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:42.613288 15567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:42.622056 15567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41321
I20260927 02:46:42.622083 15695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41321 every 8 connection(s)
I20260927 02:46:42.622992 15567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:42.625594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 15567
I20260927 02:46:42.626919 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:42.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:43323)
0927 02:46:42.625860 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.626002 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":163,"threads_started":1}
W20260927 02:46:42.627360 15696 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39687: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
I20260927 02:46:42.627956 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:42.627708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:42373)
0927 02:46:42.627814 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.627858 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:46:42.628480 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:42.628306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:47417)
0927 02:46:42.628396 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:42.628436 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 02:46:42.628651 15696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:46:43.311170 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:43.310722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:58103)
0927 02:46:43.310994 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:43.311056 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":73,"threads_started":1}
I20260927 02:46:43.445694 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:43.445276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:46061)
0927 02:46:43.445544 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:43.445590 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260927 02:46:43.629645 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:43.629214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:56991)
0927 02:46:43.629505 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:43.629555 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260927 02:46:43.821627 15703 raft_consensus.cc:482] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:43.821866 15703 raft_consensus.cc:504] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:43.823035 15703 leader_election.cc:288] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321), 311846f171af4754807cb99158716bd3 (127.0.98.2:33899)
Sep 27 02:46:43 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 02:46:43 dist-test-slave-gkrl krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 02:46:43.840245 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:43.823392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40197 (local address 127.0.98.3:41321)
0927 02:46:43.823833 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:43.823861 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:43.823930 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:46:43.824845 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:43.824854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:43.825323 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:43.825671 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:43.827509 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:43.829143 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:43.831060 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:43.831784 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:43.831972 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:43.835124 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:43.835129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:43.835140 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:43.835168 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:43.837402 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:43.837914 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:43.837923 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:43.837931 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:43.838068 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:43.838445 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:43.838451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:43.838454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:43.838977 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:43.839098 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:43.839748 (+ 650us) server_negotiation.cc:299] Negotiation successful
0927 02:46:43.840055 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":69,"threads_started":1}
I20260927 02:46:43.840242 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:43.823287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41321 (local address 127.0.98.1:40197)
0927 02:46:43.823566 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:43.823581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:43.823691 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:43.825817 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:43.825824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:43.826448 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:43.827304 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:43.827326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:43.829335 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:43.829339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:43.830928 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:43.830938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:43.831940 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:43.831945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:43.832190 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:43.832937 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:43.832950 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:43.834892 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:43.837522 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:43.837526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:43.837537 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:43.837801 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:43.838175 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:43.838178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:43.838180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:43.838346 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:43.839135 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:43.839141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:43.839383 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 02:46:43.839612 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 02:46:43.840016 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":340,"spinlock_wait_cycles":15232,"thread_start_us":60,"threads_started":1}
I20260927 02:46:43.840241 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:43.823334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33899 (local address 127.0.98.1:56767)
0927 02:46:43.823727 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:43.823744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:43.823863 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:43.825817 (+ 1954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:43.825824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:43.826448 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:43.827316 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:43.827326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:43.829336 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:43.829339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:43.830928 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:43.830938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:43.831940 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:43.831945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:43.832220 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:43.832925 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:43.832955 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:43.835634 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:43.837986 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:43.837988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:43.837990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:43.838269 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:43.838636 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:43.838638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:43.838639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:43.838713 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:43.839415 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:43.839418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:43.839490 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 02:46:43.839612 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 02:46:43.840067 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":498,"spinlock_wait_cycles":17536,"thread_start_us":96,"threads_started":1}
I20260927 02:46:43.840303 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:43.823531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56767 (local address 127.0.98.2:33899)
0927 02:46:43.823975 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:43.823994 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:43.824023 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:46:43.824845 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:43.824854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:43.825323 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:43.825671 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:43.827509 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:43.829143 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:43.831060 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:43.831784 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:43.831972 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:43.835842 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:43.835848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:43.835863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:43.835922 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:43.837887 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:43.838374 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:43.838378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:43.838382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:43.838539 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:43.838803 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:43.838806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:43.838808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:43.839301 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:43.839397 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:43.839748 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 02:46:43.840055 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":112,"threads_started":1}
W20260927 02:46:43.842692 15318 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 311846f171af4754807cb99158716bd3 (127.0.98.2:33899): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
W20260927 02:46:43.842872 15319 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
I20260927 02:46:43.842967 15319 leader_election.cc:302] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a06758cf9a5244aa986cf1523833e557; no voters: 311846f171af4754807cb99158716bd3, bf140ee16c4d433288d6e2cba4c80e95
I20260927 02:46:43.843282 15703 raft_consensus.cc:2726] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:46:44.312372 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:44.312052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:40407)
0927 02:46:44.312211 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:44.312278 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 02:46:44.446782 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:44.446378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:33355)
0927 02:46:44.446643 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:44.446696 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":107,"threads_started":1}
I20260927 02:46:44.630890 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:44.630477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:59855)
0927 02:46:44.630749 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:44.630807 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
I20260927 02:46:45.313663 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:45.313240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:35035)
0927 02:46:45.313503 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:45.313565 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260927 02:46:45.448016 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:45.447583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:40721)
0927 02:46:45.447813 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:45.447869 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260927 02:46:45.632104 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:45.631642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:51211)
0927 02:46:45.631944 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:45.632002 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260927 02:46:45.680553 15713 raft_consensus.cc:482] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:45.680732 15713 raft_consensus.cc:504] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:45.681182 15713 leader_election.cc:288] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321), 311846f171af4754807cb99158716bd3 (127.0.98.2:33899)
W20260927 02:46:45.682173 15318 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 311846f171af4754807cb99158716bd3 (127.0.98.2:33899): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
W20260927 02:46:45.682258 15319 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
I20260927 02:46:45.682302 15318 leader_election.cc:302] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a06758cf9a5244aa986cf1523833e557; no voters: 311846f171af4754807cb99158716bd3, bf140ee16c4d433288d6e2cba4c80e95
I20260927 02:46:45.682482 15713 raft_consensus.cc:2726] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:46:46.314839 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:46.314443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:40977)
0927 02:46:46.314684 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:46.314745 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260927 02:46:46.449203 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:46.448717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:55983)
0927 02:46:46.448987 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:46.449106 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":69,"threads_started":1}
I20260927 02:46:46.633440 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:46.632967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:46937)
0927 02:46:46.633289 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:46.633350 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
I20260927 02:46:47.316402 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:47.315908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:33357)
0927 02:46:47.316238 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:47.316304 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":81,"threads_started":1}
I20260927 02:46:47.450318 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:47.449906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:56593)
0927 02:46:47.450166 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:47.450228 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260927 02:46:47.634727 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:47.634288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:58187)
0927 02:46:47.634562 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:47.634626 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260927 02:46:47.979204 15720 raft_consensus.cc:482] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:47.979382 15720 raft_consensus.cc:504] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:47.979866 15720 leader_election.cc:288] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321), 311846f171af4754807cb99158716bd3 (127.0.98.2:33899)
W20260927 02:46:47.980795 15318 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 311846f171af4754807cb99158716bd3 (127.0.98.2:33899): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
W20260927 02:46:47.980904 15319 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
I20260927 02:46:47.980966 15319 leader_election.cc:302] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a06758cf9a5244aa986cf1523833e557; no voters: 311846f171af4754807cb99158716bd3, bf140ee16c4d433288d6e2cba4c80e95
I20260927 02:46:47.981101 15720 raft_consensus.cc:2726] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:46:48.317915 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:48.317438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:35909)
0927 02:46:48.317755 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:48.317818 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260927 02:46:48.451633 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:48.451108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:44119)
0927 02:46:48.451449 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:48.451538 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
I20260927 02:46:48.636222 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:48.635700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:55509)
0927 02:46:48.636039 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:48.636106 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":76,"threads_started":1}
I20260927 02:46:49.319450 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:49.318923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:41123)
0927 02:46:49.319188 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:49.319342 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260927 02:46:49.452767 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:49.452291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:58579)
0927 02:46:49.452618 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:49.452675 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":82,"threads_started":1}
I20260927 02:46:49.637369 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:49.636921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:52941)
0927 02:46:49.637208 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:49.637277 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":72,"threads_started":1}
I20260927 02:46:50.320711 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:50.320306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:47451)
0927 02:46:50.320557 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:50.320614 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260927 02:46:50.453907 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:50.453513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:55111)
0927 02:46:50.453754 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:50.453812 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260927 02:46:50.638588 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:50.638135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:51207)
0927 02:46:50.638430 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:50.638492 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260927 02:46:51.321872 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:51.321439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:59779)
0927 02:46:51.321712 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:51.321773 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260927 02:46:51.455042 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:51.454623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:35131)
0927 02:46:51.454898 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:51.454953 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20260927 02:46:51.639850 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:51.639443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:59745)
0927 02:46:51.639701 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:51.639760 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260927 02:46:52.323307 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:52.322837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:52143)
0927 02:46:52.323104 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:52.323209 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260927 02:46:52.456298 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:52.455846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:54091)
0927 02:46:52.456145 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:52.456208 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":121,"threads_started":1}
I20260927 02:46:52.641189 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:52.640707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:51059)
0927 02:46:52.640997 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:52.641070 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
I20260927 02:46:52.675806 15736 raft_consensus.cc:482] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:52.675997 15736 raft_consensus.cc:504] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:52.676465 15736 leader_election.cc:288] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321), 311846f171af4754807cb99158716bd3 (127.0.98.2:33899)
W20260927 02:46:52.677336 15319 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
W20260927 02:46:52.677439 15318 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 311846f171af4754807cb99158716bd3 (127.0.98.2:33899): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
I20260927 02:46:52.677503 15318 leader_election.cc:302] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a06758cf9a5244aa986cf1523833e557; no voters: 311846f171af4754807cb99158716bd3, bf140ee16c4d433288d6e2cba4c80e95
I20260927 02:46:52.677632 15736 raft_consensus.cc:2726] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:46:53.324661 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:53.324227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:59161)
0927 02:46:53.324483 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:53.324545 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260927 02:46:53.457692 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:53.457282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:58441)
0927 02:46:53.457542 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:53.457591 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":104,"threads_started":1}
I20260927 02:46:53.642433 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:53.642015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:49703)
0927 02:46:53.642284 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:53.642343 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
I20260927 02:46:54.325783 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:54.325385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:42475)
0927 02:46:54.325628 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:54.325691 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260927 02:46:54.458889 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:54.458448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:55019)
0927 02:46:54.458736 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:54.458796 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
I20260927 02:46:54.643760 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:54.643242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:42015)
0927 02:46:54.643571 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:54.643649 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260927 02:46:55.326992 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:55.326538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:52441)
0927 02:46:55.326838 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:55.326893 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260927 02:46:55.460183 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:55.459662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:54603)
0927 02:46:55.460016 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:55.460081 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":87,"threads_started":1}
I20260927 02:46:55.645066 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:55.644636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:43597)
0927 02:46:55.644912 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:55.644969 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260927 02:46:56.328397 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:56.327951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:47045)
0927 02:46:56.328236 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:56.328302 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
I20260927 02:46:56.461323 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:56.460932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:54543)
0927 02:46:56.461165 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:56.461229 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260927 02:46:56.555641 15748 raft_consensus.cc:482] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:56.555804 15748 raft_consensus.cc:504] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf140ee16c4d433288d6e2cba4c80e95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "311846f171af4754807cb99158716bd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "a06758cf9a5244aa986cf1523833e557" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44189 } }
I20260927 02:46:56.556329 15748 leader_election.cc:288] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321), 311846f171af4754807cb99158716bd3 (127.0.98.2:33899)
W20260927 02:46:56.557183 15319 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf140ee16c4d433288d6e2cba4c80e95 (127.0.98.3:41321): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
W20260927 02:46:56.557314 15318 leader_election.cc:341] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 311846f171af4754807cb99158716bd3 (127.0.98.2:33899): Not found: Tablet not found: ac188b6bcf984856b2a3685e72db9b55
I20260927 02:46:56.557410 15318 leader_election.cc:302] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a06758cf9a5244aa986cf1523833e557; no voters: 311846f171af4754807cb99158716bd3, bf140ee16c4d433288d6e2cba4c80e95
I20260927 02:46:56.557588 15748 raft_consensus.cc:2726] T ac188b6bcf984856b2a3685e72db9b55 P a06758cf9a5244aa986cf1523833e557 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:46:56.646245 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:56.645827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:54799)
0927 02:46:56.646090 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:56.646143 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260927 02:46:57.329779 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:57.329337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:42371)
0927 02:46:57.329623 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:57.329684 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260927 02:46:57.462729 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:57.462294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:42217)
0927 02:46:57.462558 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:57.462630 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260927 02:46:57.627350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 15266
I20260927 02:46:57.633486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 15434
I20260927 02:46:57.639000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 15567
2026-09-27T02:46:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19077 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:57 dist-test-slave-gkrl krb5kdc[15754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:57 dist-test-slave-gkrl krb5kdc[15754](info): set up 2 sockets
Sep 27 02:46:57 dist-test-slave-gkrl krb5kdc[15754](info): commencing operation
krb5kdc: starting...
W20260927 02:46:58.692381 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:58 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:58.812373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41909
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:58.922459 15767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:58.922772 15767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:58.922851 15767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:58.926877 15767 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:46:58.926959 15767 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:46:58.926985 15767 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:46:58.927016 15767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:58.927057 15767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:58.927098 15767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:58.927134 15767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:58.927173 15767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:58.932323 15767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:58.933352 15767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:58.934357 15767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:46:58.943032 15767 server_base.cc:1034] running on GCE node
W20260927 02:46:58.942970 15772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:58.943140 15773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:58.942955 15775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:58.944018 15767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:58.945123 15767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:58.946311 15767 hybrid_clock.cc:648] HybridClock initialized: now 1790477218946278 us; error 49 us; skew 500 ppm
Sep 27 02:46:58 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:58.949363 15767 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:58.950250 15767 webserver.cc:460] Webserver started at http://127.0.98.62:33735/ using document root <none> and password file <none>
I20260927 02:46:58.950799 15767 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:58.950887 15767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:58.951112 15767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:58.953186 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "05713974609043c69425adb1f2f2b824"
format_stamp: "Formatted at 2026-09-27 02:46:58 on dist-test-slave-gkrl"
server_key: "436fbab39fdaaaa1ed7dd21c49766560"
server_key_iv: "ad94c86b33f623be7345cd315760f527"
server_key_version: "encryptionkey@0"
I20260927 02:46:58.953754 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "05713974609043c69425adb1f2f2b824"
format_stamp: "Formatted at 2026-09-27 02:46:58 on dist-test-slave-gkrl"
server_key: "436fbab39fdaaaa1ed7dd21c49766560"
server_key_iv: "ad94c86b33f623be7345cd315760f527"
server_key_version: "encryptionkey@0"
I20260927 02:46:58.957424 15767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:46:58.959844 15782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:58.961148 15767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:58.961274 15767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "05713974609043c69425adb1f2f2b824"
format_stamp: "Formatted at 2026-09-27 02:46:58 on dist-test-slave-gkrl"
server_key: "436fbab39fdaaaa1ed7dd21c49766560"
server_key_iv: "ad94c86b33f623be7345cd315760f527"
server_key_version: "encryptionkey@0"
I20260927 02:46:58.961386 15767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:58.986207 15767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:58.989857 15767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:58.990078 15767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:58.997115 15767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41909
I20260927 02:46:58.997416 15834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41909 every 8 connection(s)
I20260927 02:46:58.998124 15767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:58.998489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 15767
I20260927 02:46:58.998735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:58.999053 392 external_mini_cluster.cc:1428] Setting key 69459099b5f0808bc757f836635c4f4a
I20260927 02:46:59.001199 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:59.007849 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Bootstrap starting.
Sep 27 02:46:59 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:59.010607 15835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:59.011471 15835 log.cc:824] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:59.013692 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: No bootstrap required, opened a new log
I20260927 02:46:59.013875 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.000541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.0.98.62:41909)
0927 02:46:59.000959 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.000980 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.001015 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:46:59.001717 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.001734 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.002053 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.002333 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:59.003232 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.004312 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.005434 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.005888 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.005998 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.008866 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:59.008872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:59.008890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:59.008930 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:59.010872 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.011308 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.011316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.011323 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.011461 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.011771 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.011774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.011777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.012233 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:59.012459 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.012671 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.012964 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
I20260927 02:46:59.017427 15835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:46:59.017639 15835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:59.017675 15835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05713974609043c69425adb1f2f2b824, State: Initialized, Role: FOLLOWER
I20260927 02:46:59.018014 15835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:46:59.018123 15835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:59.018162 15835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:59.018222 15835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:59.019310 15835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:46:59.019635 15835 leader_election.cc:302] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05713974609043c69425adb1f2f2b824; no voters:
I20260927 02:46:59.019989 15835 leader_election.cc:288] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:59.020219 15840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:59.020493 15840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 1 LEADER]: Becoming Leader. State: Replica: 05713974609043c69425adb1f2f2b824, State: Running, Role: LEADER
I20260927 02:46:59.020773 15840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:46:59.021203 15835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:59.022333 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05713974609043c69425adb1f2f2b824. Latest consensus state: current_term: 1 leader_uuid: "05713974609043c69425adb1f2f2b824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } } }
I20260927 02:46:59.022464 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:59.022848 15849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:59.022872 15841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05713974609043c69425adb1f2f2b824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } } }
I20260927 02:46:59.023579 15841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:59.025517 15849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:59.030869 15849 catalog_manager.cc:1347] Generated new cluster ID: 66fc94ead515456eb16fd1eb51400168
I20260927 02:46:59.030932 15849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:59.073815 15849 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:59.074932 15849 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:59.084692 15849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Generated new TSK 0
I20260927 02:46:59.085470 15849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:46:59.116503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:59.225930 15861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:59.226248 15861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:59.226332 15861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:59.230331 15861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:59.230409 15861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:59.230491 15861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:59.235661 15861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:59.236651 15861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:59.237627 15861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:59.247306 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:59.247382 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:59.247306 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:59.247532 15861 server_base.cc:1034] running on GCE node
I20260927 02:46:59.248054 15861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:59.248664 15861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:59.249855 15861 hybrid_clock.cc:648] HybridClock initialized: now 1790477219249815 us; error 54 us; skew 500 ppm
Sep 27 02:46:59 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:59.252734 15861 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:59.253543 15861 webserver.cc:460] Webserver started at http://127.0.98.1:33085/ using document root <none> and password file <none>
I20260927 02:46:59.254078 15861 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:59.254204 15861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:59.254421 15861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:59.256405 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "51a983a30eda4b2584783d6af62cefd9"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "79e529c3915be5780954676e7675d7bb"
server_key_iv: "729306ae2712850cc1f8e087bbb78a17"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.256935 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "51a983a30eda4b2584783d6af62cefd9"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "79e529c3915be5780954676e7675d7bb"
server_key_iv: "729306ae2712850cc1f8e087bbb78a17"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.260573 15861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:46:59.263002 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.264297 15861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:46:59.264431 15861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "51a983a30eda4b2584783d6af62cefd9"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "79e529c3915be5780954676e7675d7bb"
server_key_iv: "729306ae2712850cc1f8e087bbb78a17"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.264572 15861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:59.276975 15861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:59.280097 15861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:59.280298 15861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:59.280879 15861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:59.281683 15861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:59.281752 15861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.281821 15861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:59.281867 15861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.290930 15861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39659
I20260927 02:46:59.290951 15989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39659 every 8 connection(s)
I20260927 02:46:59.291826 15861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:59.292454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 15861
I20260927 02:46:59.292577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:46:59.292876 392 external_mini_cluster.cc:1428] Setting key 53cf03e9bb71cf52237e4d445c5ffd91
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 02:46:59 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:46:59.306538 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.293712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59797 (local address 127.0.98.62:41909)
0927 02:46:59.293847 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.293856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.295344 (+ 1488us) server_negotiation.cc:407] Connection header received
0927 02:46:59.296567 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.296570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.296618 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.296684 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:59.298006 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.298561 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.299738 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.300054 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.300169 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.302689 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:59.302693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:59.302696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:59.302722 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:59.304285 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.304858 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.304862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.304863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.304946 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.305334 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.305337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.305339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.305528 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:59.305637 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.306300 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.306414 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:46:59.307600 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.294229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:59797)
0927 02:46:59.295166 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:59.295219 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:59.296340 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:59.296902 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:59.296914 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:59.297373 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:59.297860 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.297873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.298693 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.298696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.299594 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.299603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.300208 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.300212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.300392 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.301012 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:59.301028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:59.302520 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:59.304417 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:59.304423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:59.304439 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:59.304751 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:59.305046 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:59.305049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:59.305051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:59.305221 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:59.305637 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:59.305645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:59.306012 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 02:46:59.306254 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 02:46:59.306635 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":582,"threads_started":1}
I20260927 02:46:59.309021 15990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260927 02:46:59.309332 15990 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:59.309835 15990 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260927 02:46:59.311355 15799 ts_manager.cc:194] Registered new tserver with Master: 51a983a30eda4b2584783d6af62cefd9 (127.0.98.1:39659)
I20260927 02:46:59.312525 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:46:59.319480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:59.428299 15995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:59.428574 15995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:59.428648 15995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:59.432587 15995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:59.432665 15995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:59.432758 15995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:59.437752 15995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:59.438751 15995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:59.439749 15995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:59.449084 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:59.449074 16003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:59.449159 16000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:59.449225 15995 server_base.cc:1034] running on GCE node
I20260927 02:46:59.449834 15995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:59.450500 15995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:59.451684 15995 hybrid_clock.cc:648] HybridClock initialized: now 1790477219451676 us; error 33 us; skew 500 ppm
Sep 27 02:46:59 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:59.454360 15995 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:59.455186 15995 webserver.cc:460] Webserver started at http://127.0.98.2:41417/ using document root <none> and password file <none>
I20260927 02:46:59.455677 15995 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:59.455731 15995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:59.455941 15995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:59.457960 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ee4d138614a4400b2f5681b40850016"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "566d8ca95092e3b31a735f5557ce6d55"
server_key_iv: "5c4afd9cbdea4eeaadf90e8df9682d66"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.458518 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ee4d138614a4400b2f5681b40850016"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "566d8ca95092e3b31a735f5557ce6d55"
server_key_iv: "5c4afd9cbdea4eeaadf90e8df9682d66"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.462224 15995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:46:59.464672 16010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.466003 15995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:59.466094 15995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "2ee4d138614a4400b2f5681b40850016"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "566d8ca95092e3b31a735f5557ce6d55"
server_key_iv: "5c4afd9cbdea4eeaadf90e8df9682d66"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.466241 15995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:59.496809 15995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:59.499902 15995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:59.500056 15995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:59.500588 15995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:59.501363 15995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:59.501408 15995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.501447 15995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:59.501462 15995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.510097 15995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45947
I20260927 02:46:59.510214 16123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45947 every 8 connection(s)
I20260927 02:46:59.511020 15995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:59.515841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 15995
I20260927 02:46:59.515986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:59.516258 392 external_mini_cluster.cc:1428] Setting key 7c47a6837ab8c9993059757f7de4477f
Sep 27 02:46:59 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:46:59.525734 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.512867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52673 (local address 127.0.98.62:41909)
0927 02:46:59.513030 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.513035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.513887 (+ 852us) server_negotiation.cc:407] Connection header received
0927 02:46:59.515180 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.515184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.515243 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.515341 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:59.516638 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.517251 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.518482 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.518872 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.518991 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.521936 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:59.521939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:59.521942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:59.521966 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:59.523575 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.524209 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.524214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.524216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.524310 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.524673 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.524676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.524677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.524856 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:59.524965 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.525443 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.525568 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:46:59.526494 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.513287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:52673)
0927 02:46:59.513748 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:59.513790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:59.514959 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:59.515510 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:59.515516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:59.515857 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:59.516499 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.516515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.517419 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.517423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.518356 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.518366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.519097 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.519104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.519345 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.519982 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:59.520006 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:59.521760 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:59.523719 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:59.523725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:59.523739 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:59.524102 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:59.524403 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:59.524408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:59.524410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:59.524572 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:59.524963 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:59.524970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:59.525208 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 02:46:59.525386 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 02:46:59.525700 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260927 02:46:59.527753 16124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260927 02:46:59.528019 16124 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:46:59.528548 16124 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260927 02:46:59.529596 15799 ts_manager.cc:194] Registered new tserver with Master: 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
I20260927 02:46:59.530151 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:46:59.543354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:59.653934 16129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:59.654160 16129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:59.654201 16129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:59.658144 16129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:59.658195 16129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:59.658250 16129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:59.663290 16129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:46:59.664294 16129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:59.665346 16129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:59.675506 16135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:59.675506 16134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:59.675544 16137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:59.675719 16129 server_base.cc:1034] running on GCE node
I20260927 02:46:59.676227 16129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:59.676865 16129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:59.678036 16129 hybrid_clock.cc:648] HybridClock initialized: now 1790477219678005 us; error 43 us; skew 500 ppm
Sep 27 02:46:59 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:59.680792 16129 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:59.681609 16129 webserver.cc:460] Webserver started at http://127.0.98.3:39075/ using document root <none> and password file <none>
I20260927 02:46:59.682120 16129 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:59.682176 16129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:59.682333 16129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:59.684417 16129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "ec6216da7fa042aaaead9e00c56092ce"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "c4ce21621329b2295b1e11cef80e795d"
server_key_iv: "81605e4c482f8c1d233b057c1c036a32"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.684929 16129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ec6216da7fa042aaaead9e00c56092ce"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "c4ce21621329b2295b1e11cef80e795d"
server_key_iv: "81605e4c482f8c1d233b057c1c036a32"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.689193 16129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 02:46:59.691552 16144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.692934 16129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:59.693111 16129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "ec6216da7fa042aaaead9e00c56092ce"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "c4ce21621329b2295b1e11cef80e795d"
server_key_iv: "81605e4c482f8c1d233b057c1c036a32"
server_key_version: "encryptionkey@0"
I20260927 02:46:59.693223 16129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:59.711922 16129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:59.714684 16129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:59.714879 16129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:59.715453 16129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:59.716315 16129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:59.716361 16129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.716423 16129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:59.716456 16129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:59.725574 16129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40247
I20260927 02:46:59.725608 16257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40247 every 8 connection(s)
I20260927 02:46:59.726413 16129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:59.729616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 16129
I20260927 02:46:59.729748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:59.730024 392 external_mini_cluster.cc:1428] Setting key eee40b483903980371343be4d2245377
Sep 27 02:46:59 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:59.739319 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.728172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56207 (local address 127.0.98.62:41909)
0927 02:46:59.728301 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.728305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.729341 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 02:46:59.730428 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.730431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.730477 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.730603 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:59.731828 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.732467 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.733565 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.733777 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.733859 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.736199 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:59.736202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:59.736204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:59.736226 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:59.737505 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.737981 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.737984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.737986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.738063 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.738409 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.738411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.738413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.738577 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:59.738663 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.739095 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.739201 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:46:59.740181 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.728588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:56207)
0927 02:46:59.729199 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:59.729237 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:59.730200 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:59.730745 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:59.730752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:59.731199 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:59.731702 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.731715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.732575 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.732578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.733441 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.733450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.733904 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.733906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.734038 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.734605 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:59.734621 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:59.736059 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:59.737614 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:59.737619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:59.737640 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:59.737872 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:59.738178 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:59.738180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:59.738181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:59.738320 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:59.738681 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:59.738686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:59.738901 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 02:46:59.739054 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 02:46:59.739285 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":110,"threads_started":1}
I20260927 02:46:59.741304 16258 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260927 02:46:59.741546 16258 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:59.742002 16258 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260927 02:46:59.743077 15799 ts_manager.cc:194] Registered new tserver with Master: ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247)
I20260927 02:46:59.743721 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56207
I20260927 02:46:59.743916 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:59.753106 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.745946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.0.98.62:41909)
0927 02:46:59.746104 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.746109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.746266 (+ 157us) server_negotiation.cc:407] Connection header received
0927 02:46:59.746455 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.746459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.746500 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.746591 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:59.747315 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.747811 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.748663 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.748879 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.748984 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.750264 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:59.750268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:59.750270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:59.750301 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:59.751484 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.751917 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.751923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.751926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.752003 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:59.752272 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:59.752276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:59.752278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:59.752478 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:59.752618 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.752843 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.752954 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 02:46:59.755947 15799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:59.758872 15799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:59.773455 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.767277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40247 (local address 127.0.0.1:59970)
0927 02:46:59.767756 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:59.767801 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:59.767944 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:59.769006 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:59.769011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:59.769033 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:59.769194 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.769201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.770863 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.770866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.772023 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.772033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.773053 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.773057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.773155 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.773183 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:46:59.773235 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:46:59.773293 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":125,"thread_start_us":68,"threads_started":1}
I20260927 02:46:59.773490 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.767443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.0.98.3:40247)
0927 02:46:59.768529 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.768541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.768563 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:46:59.768637 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.768641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.768838 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.769339 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:59.769500 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.770712 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.772184 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.772953 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.773065 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.773215 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.773264 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.773349 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":165,"threads_started":1}
I20260927 02:46:59.773875 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.768204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.98.2:45947)
0927 02:46:59.768975 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.768988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.769010 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:46:59.769081 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.769087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.769273 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.769414 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:59.769721 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.770480 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.772142 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.772684 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.772807 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.772962 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.773372 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.773460 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":161,"threads_started":1}
I20260927 02:46:59.774405 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.767504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:36852)
0927 02:46:59.768199 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:59.768212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:59.768287 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:59.769177 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:59.769180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:59.769194 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:59.769339 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.769346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.770863 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.770866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.772977 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.772990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.774038 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.774041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.774126 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.774149 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:46:59.774195 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:46:59.774254 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":834,"thread_start_us":99,"threads_started":1}
I20260927 02:46:59.774314 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.768610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45947 (local address 127.0.0.1:40246)
0927 02:46:59.768864 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:59.768876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:59.768957 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:59.769396 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:59.769399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:59.769416 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:59.769511 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.769517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.770748 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.770755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.772027 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.772036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.772804 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.772808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.773020 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.773067 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 02:46:59.773172 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 02:46:59.773293 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":243,"thread_start_us":116,"threads_started":1}
I20260927 02:46:59.776738 16192 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4d7f145d0f4a58a7563844b350f1b1 (DEFAULT_TABLE table=test-table [id=a90c2fdaf2994fb184a9caf3ff63503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:59.777330 16058 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4d7f145d0f4a58a7563844b350f1b1 (DEFAULT_TABLE table=test-table [id=a90c2fdaf2994fb184a9caf3ff63503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:59.777880 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4d7f145d0f4a58a7563844b350f1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:59.778342 16058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4d7f145d0f4a58a7563844b350f1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:59.780216 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.768261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.0.98.1:39659)
0927 02:46:59.768654 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.768670 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.768699 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:46:59.768781 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.768786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.768975 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.769202 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:59.769464 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.770710 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.773108 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.773907 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.774383 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.774510 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.780002 (+ 5492us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.780103 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":113,"threads_started":1}
I20260927 02:46:59.782557 15924 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4d7f145d0f4a58a7563844b350f1b1 (DEFAULT_TABLE table=test-table [id=a90c2fdaf2994fb184a9caf3ff63503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:59.783541 15924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4d7f145d0f4a58a7563844b350f1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:59.785181 16275 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Bootstrap starting.
I20260927 02:46:59.785579 16276 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Bootstrap starting.
I20260927 02:46:59.787822 16276 tablet_bootstrap.cc:654] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:59.788046 16275 tablet_bootstrap.cc:654] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:59.788913 16275 log.cc:824] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:59.789100 16276 log.cc:824] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:59.790361 16279 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Bootstrap starting.
I20260927 02:46:59.790913 16275 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: No bootstrap required, opened a new log
I20260927 02:46:59.791035 16276 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: No bootstrap required, opened a new log
I20260927 02:46:59.791160 16275 ts_tablet_manager.cc:1397] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:46:59.791230 16276 ts_tablet_manager.cc:1397] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 02:46:59.792542 16279 tablet_bootstrap.cc:654] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:59.793761 16279 log.cc:824] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:59.794191 16275 raft_consensus.cc:348] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:46:59.794507 16275 raft_consensus.cc:374] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:59.794566 16275 raft_consensus.cc:729] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee4d138614a4400b2f5681b40850016, State: Initialized, Role: FOLLOWER
I20260927 02:46:59.795049 16275 consensus_queue.cc:260] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:46:59.795166 16276 raft_consensus.cc:348] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:46:59.795398 16276 raft_consensus.cc:374] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:59.795445 16276 raft_consensus.cc:729] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6216da7fa042aaaead9e00c56092ce, State: Initialized, Role: FOLLOWER
I20260927 02:46:59.795784 16279 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: No bootstrap required, opened a new log
I20260927 02:46:59.795984 16279 ts_tablet_manager.cc:1397] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:46:59.795925 16276 consensus_queue.cc:260] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:46:59.796111 16275 ts_tablet_manager.cc:1428] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:46:59.796355 16124 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
I20260927 02:46:59.796825 16276 ts_tablet_manager.cc:1428] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:46:59.797153 16258 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
I20260927 02:46:59.799963 16279 raft_consensus.cc:348] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:46:59.800230 16279 raft_consensus.cc:374] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:59.800307 16279 raft_consensus.cc:729] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a983a30eda4b2584783d6af62cefd9, State: Initialized, Role: FOLLOWER
I20260927 02:46:59.800761 16279 consensus_queue.cc:260] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:46:59.801369 15990 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
I20260927 02:46:59.801669 16279 ts_tablet_manager.cc:1428] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 02:46:59.977339 16259 tablet.cc:2369] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:59.986447 16283 raft_consensus.cc:482] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:59.986666 16283 raft_consensus.cc:504] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:46:59.987804 16283 leader_election.cc:288] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
I20260927 02:46:59.993570 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.988128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40247 (local address 127.0.98.1:49467)
0927 02:46:59.988684 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:59.988696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:59.988791 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:59.989065 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:59.989068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:59.989234 (+ 166us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:59.989452 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.989460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.990527 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.990531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.992200 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.992211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.993133 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.993138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.993237 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.993260 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:46:59.993327 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:46:59.993387 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":275,"spinlock_wait_cycles":287104,"thread_start_us":112,"threads_started":1}
I20260927 02:46:59.993593 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.988290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49467 (local address 127.0.98.3:40247)
0927 02:46:59.988431 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.988435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.988797 (+ 362us) server_negotiation.cc:407] Connection header received
0927 02:46:59.988861 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.988863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.988909 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.989006 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:59.989595 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.990378 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.992349 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.992993 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.993114 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.993189 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.993374 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.993450 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 02:46:59.993767 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.988128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45947 (local address 127.0.98.1:43363)
0927 02:46:59.988574 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:59.988590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:59.988719 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:59.989065 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:59.989069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:59.989226 (+ 157us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:59.989454 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.989460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.990527 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.990531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.992204 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:59.992211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.993272 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.993278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:59.993427 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.993450 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:46:59.993530 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:46:59.993607 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":191,"spinlock_wait_cycles":28672,"thread_start_us":128,"threads_started":1}
I20260927 02:46:59.993782 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:59.988257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43363 (local address 127.0.98.2:45947)
0927 02:46:59.988397 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:59.988403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:59.988682 (+ 279us) server_negotiation.cc:407] Connection header received
0927 02:46:59.988852 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:59.988855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:59.988911 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:59.989008 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:59.989595 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.990411 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.992349 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:59.992995 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:59.993171 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:59.993241 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:59.993578 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:46:59.993664 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:46:59.994458 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4d7f145d0f4a58a7563844b350f1b1" candidate_uuid: "51a983a30eda4b2584783d6af62cefd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6216da7fa042aaaead9e00c56092ce" is_pre_election: true
I20260927 02:46:59.994474 16078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4d7f145d0f4a58a7563844b350f1b1" candidate_uuid: "51a983a30eda4b2584783d6af62cefd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee4d138614a4400b2f5681b40850016" is_pre_election: true
I20260927 02:46:59.994784 16078 raft_consensus.cc:2454] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a983a30eda4b2584783d6af62cefd9 in term 0.
I20260927 02:46:59.994793 16212 raft_consensus.cc:2454] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a983a30eda4b2584783d6af62cefd9 in term 0.
I20260927 02:46:59.995280 15880 leader_election.cc:302] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ee4d138614a4400b2f5681b40850016, 51a983a30eda4b2584783d6af62cefd9, ec6216da7fa042aaaead9e00c56092ce; no voters:
I20260927 02:46:59.995582 16283 raft_consensus.cc:2781] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:59.995666 16283 raft_consensus.cc:482] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:59.995720 16283 raft_consensus.cc:3037] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:59.997035 16283 raft_consensus.cc:504] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:46:59.997453 16283 leader_election.cc:288] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 1 election: Requested vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
I20260927 02:46:59.997901 16078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4d7f145d0f4a58a7563844b350f1b1" candidate_uuid: "51a983a30eda4b2584783d6af62cefd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee4d138614a4400b2f5681b40850016"
I20260927 02:46:59.998037 16078 raft_consensus.cc:3037] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:59.999570 16078 raft_consensus.cc:2454] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a983a30eda4b2584783d6af62cefd9 in term 1.
I20260927 02:46:59.999962 15880 leader_election.cc:302] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ee4d138614a4400b2f5681b40850016, 51a983a30eda4b2584783d6af62cefd9; no voters:
I20260927 02:47:00.000175 16283 raft_consensus.cc:2781] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:00.000443 16283 raft_consensus.cc:686] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 1 LEADER]: Becoming Leader. State: Replica: 51a983a30eda4b2584783d6af62cefd9, State: Running, Role: LEADER
I20260927 02:47:00.000773 16283 consensus_queue.cc:237] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:47:00.001047 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4d7f145d0f4a58a7563844b350f1b1" candidate_uuid: "51a983a30eda4b2584783d6af62cefd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6216da7fa042aaaead9e00c56092ce"
I20260927 02:47:00.001183 16212 raft_consensus.cc:3037] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:00.002483 16212 raft_consensus.cc:2454] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a983a30eda4b2584783d6af62cefd9 in term 1.
I20260927 02:47:00.004554 15799 catalog_manager.cc:5495] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a983a30eda4b2584783d6af62cefd9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "51a983a30eda4b2584783d6af62cefd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } health_report { overall_health: UNKNOWN } } }
W20260927 02:47:00.012156 16125 tablet.cc:2369] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:47:00.043275 15991 tablet.cc:2369] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:47:00.064213 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:00.060451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.0.98.1:39659)
0927 02:47:00.060627 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:00.060632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:00.060647 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:47:00.060845 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:00.060848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:00.060908 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:00.061029 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:00.061321 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:00.061833 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:00.062698 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:00.062921 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:00.063031 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:00.063337 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:00.063765 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:00.063923 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:00.064010 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 02:47:00.064067 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 02:47:00.070272 16212 raft_consensus.cc:1261] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 1 FOLLOWER]: Refusing update from remote peer 51a983a30eda4b2584783d6af62cefd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:00.070272 16078 raft_consensus.cc:1261] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 1 FOLLOWER]: Refusing update from remote peer 51a983a30eda4b2584783d6af62cefd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:00.070981 16283 consensus_queue.cc:1035] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:00.071254 16286 consensus_queue.cc:1035] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:00.079988 16291 mvcc.cc:205] Tried to move back new op lower bound from 7333794693400588288 to 7333794693126860800. Current Snapshot: MvccSnapshot[applied={T|T < 7333794693400588288}]
I20260927 02:47:00.080565 16289 mvcc.cc:205] Tried to move back new op lower bound from 7333794693400588288 to 7333794693126860800. Current Snapshot: MvccSnapshot[applied={T|T < 7333794693400588288 or (T in {7333794693400588288})}]
I20260927 02:47:00.087239 15797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:00.087431 15797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:00.089978 15797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Sending DeleteTablet for 3 replicas of tablet 6c4d7f145d0f4a58a7563844b350f1b1
I20260927 02:47:00.090835 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4d7f145d0f4a58a7563844b350f1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59970
I20260927 02:47:00.090889 16058 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4d7f145d0f4a58a7563844b350f1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40246
I20260927 02:47:00.090889 15924 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4d7f145d0f4a58a7563844b350f1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36852
I20260927 02:47:00.091462 16301 tablet_replica.cc:330] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: stopping tablet replica
I20260927 02:47:00.091706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 15861
I20260927 02:47:00.091833 16301 raft_consensus.cc:2229] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:00.092489 16301 raft_consensus.cc:2258] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:00.093017 16299 tablet_replica.cc:330] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: stopping tablet replica
I20260927 02:47:00.093312 16299 raft_consensus.cc:2229] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:00.093600 16299 raft_consensus.cc:2258] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:00.095031 16299 ts_tablet_manager.cc:1905] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:00.095031 16301 ts_tablet_manager.cc:1905] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:00.097759 16301 ts_tablet_manager.cc:1918] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:00.097759 16299 ts_tablet_manager.cc:1918] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:00.097862 16301 log.cc:1196] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/6c4d7f145d0f4a58a7563844b350f1b1
I20260927 02:47:00.097862 16299 log.cc:1196] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/6c4d7f145d0f4a58a7563844b350f1b1
I20260927 02:47:00.098197 16301 ts_tablet_manager.cc:1939] T 6c4d7f145d0f4a58a7563844b350f1b1 P ec6216da7fa042aaaead9e00c56092ce: Deleting consensus metadata
I20260927 02:47:00.098202 16299 ts_tablet_manager.cc:1939] T 6c4d7f145d0f4a58a7563844b350f1b1 P 2ee4d138614a4400b2f5681b40850016: Deleting consensus metadata
W20260927 02:47:00.098860 15786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39659 (error 108)
I20260927 02:47:00.099399 15786 catalog_manager.cc:4841] TS 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947): tablet 6c4d7f145d0f4a58a7563844b350f1b1 (table test-table [id=a90c2fdaf2994fb184a9caf3ff63503a]) successfully deleted
I20260927 02:47:00.099632 15783 catalog_manager.cc:4841] TS ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247): tablet 6c4d7f145d0f4a58a7563844b350f1b1 (table test-table [id=a90c2fdaf2994fb184a9caf3ff63503a]) successfully deleted
I20260927 02:47:00.099741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 15995
I20260927 02:47:00.100575 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:00.099714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:36866)
0927 02:47:00.100394 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:00.100474 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578}
W20260927 02:47:00.101198 15786 catalog_manager.cc:4584] TS 51a983a30eda4b2584783d6af62cefd9 (127.0.98.1:39659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c4d7f145d0f4a58a7563844b350f1b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
I20260927 02:47:00.105453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 16129
I20260927 02:47:00.110985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 15767
I20260927 02:47:20.117856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41909
--webserver_interface=127.0.98.62
--webserver_port=33735
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:20.226429 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:20.226734 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:20.226812 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:20.230772 16303 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:47:20.230847 16303 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:47:20.230876 16303 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:47:20.230911 16303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:20.230948 16303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:20.231005 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:20.231051 16303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:20.231086 16303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:20.236245 16303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:20.237262 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:20.238255 16303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:20.252228 16312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:20.252192 16310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:20.252192 16309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:20.252552 16303 server_base.cc:1034] running on GCE node
I20260927 02:47:20.253305 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:20.254489 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:20.255694 16303 hybrid_clock.cc:648] HybridClock initialized: now 1790477240255674 us; error 33 us; skew 500 ppm
Sep 27 02:47:20 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:20.258601 16303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:20.259491 16303 webserver.cc:460] Webserver started at http://127.0.98.62:33735/ using document root <none> and password file <none>
I20260927 02:47:20.260097 16303 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:20.260169 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:20.264333 16303 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:47:20.266513 16319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.267750 16303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:20.267879 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "05713974609043c69425adb1f2f2b824"
format_stamp: "Formatted at 2026-09-27 02:46:58 on dist-test-slave-gkrl"
server_key: "436fbab39fdaaaa1ed7dd21c49766560"
server_key_iv: "ad94c86b33f623be7345cd315760f527"
server_key_version: "encryptionkey@0"
I20260927 02:47:20.268488 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:20.289760 16303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:20.292869 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:20.293063 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:20.300459 16303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41909
I20260927 02:47:20.300791 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41909 every 8 connection(s)
I20260927 02:47:20.301597 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:20.304147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 16303
I20260927 02:47:20.304605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39659
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41909
--webserver_port=33085
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:47:20.306118 16372 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:47:20.307744 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Bootstrap starting.
I20260927 02:47:20.324539 16372 log.cc:824] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:20.333113 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:47:20.333479 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Bootstrap complete.
I20260927 02:47:20.337528 16372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:47:20.337934 16372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05713974609043c69425adb1f2f2b824, State: Initialized, Role: FOLLOWER
I20260927 02:47:20.338443 16372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:47:20.338589 16372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:20.338678 16372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:20.338797 16372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:47:20.340646 16372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:47:20.340966 16372 leader_election.cc:302] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05713974609043c69425adb1f2f2b824; no voters:
I20260927 02:47:20.341315 16372 leader_election.cc:288] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:47:20.341391 16376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:47:20.341661 16376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 2 LEADER]: Becoming Leader. State: Replica: 05713974609043c69425adb1f2f2b824, State: Running, Role: LEADER
I20260927 02:47:20.341995 16376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:47:20.342559 16372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:20.342870 16376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05713974609043c69425adb1f2f2b824. Latest consensus state: current_term: 2 leader_uuid: "05713974609043c69425adb1f2f2b824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } } }
I20260927 02:47:20.342986 16376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:20.343350 16377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05713974609043c69425adb1f2f2b824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } } }
I20260927 02:47:20.343549 16377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:20.343330 16380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:20.348435 16380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:20.349112 16380 catalog_manager.cc:1259] Loaded cluster ID: 66fc94ead515456eb16fd1eb51400168
I20260927 02:47:20.349172 16380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:20.351478 16380 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:20.352229 16380 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Loaded TSK: 0
I20260927 02:47:20.367493 16380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Generated new TSK 1
I20260927 02:47:20.367738 16380 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:47:20.419026 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:20.419256 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:20.419296 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:20.423250 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:20.423302 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:20.423357 16374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:20.428531 16374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33085
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:20.429446 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:20.430405 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:20.440857 16399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:20.441123 16374 server_base.cc:1034] running on GCE node
W20260927 02:47:20.440862 16400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:20.440886 16402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:20.441713 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:20.442435 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:20.443667 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790477240443643 us; error 45 us; skew 500 ppm
Sep 27 02:47:20 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:20.446493 16374 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:20.447396 16374 webserver.cc:460] Webserver started at http://127.0.98.1:33085/ using document root <none> and password file <none>
I20260927 02:47:20.447988 16374 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:20.448087 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:20.451973 16374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:20.454169 16409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.455427 16374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:20.455561 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "51a983a30eda4b2584783d6af62cefd9"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "79e529c3915be5780954676e7675d7bb"
server_key_iv: "729306ae2712850cc1f8e087bbb78a17"
server_key_version: "encryptionkey@0"
I20260927 02:47:20.456135 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:20.465790 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:20.468807 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:20.469004 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:20.469550 16374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:20.470644 16416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:47:20.475775 16374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:47:20.475857 16374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 02:47:20.475960 16374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:47:20.477546 16374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:47:20.477614 16374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:47:20.477916 16416 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Bootstrap starting.
I20260927 02:47:20.487963 16374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39659
I20260927 02:47:20.488168 16523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39659 every 8 connection(s)
I20260927 02:47:20.488919 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:20.490592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 16374
I20260927 02:47:20.491083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45947
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41909
--webserver_port=41417
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:47:20.499003 16416 log.cc:824] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:47:20 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:20.507241 16416 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:47:20.507660 16416 tablet_bootstrap.cc:492] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Bootstrap complete.
I20260927 02:47:20.508158 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.490946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:58485)
0927 02:47:20.492901 (+ 1955us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:20.492963 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:20.494101 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:20.495344 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:20.495351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:20.495775 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:20.496361 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.496377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.497618 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.497627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.498473 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.498483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.499240 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.499244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.499430 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.500169 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:20.500188 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:20.502189 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:20.504786 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.504792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.504805 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.505153 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.505573 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.505576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.505578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.505743 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.506414 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:20.506421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:20.506722 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 02:47:20.506906 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 02:47:20.507229 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1705,"thread_start_us":143,"threads_started":1}
I20260927 02:47:20.508678 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.492193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58485 (local address 127.0.98.62:41909)
0927 02:47:20.494244 (+ 2051us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.494264 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.494309 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:47:20.494552 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.494562 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.494921 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.495291 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:20.496525 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.497466 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.498601 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.499062 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.499281 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.502407 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:20.502413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:20.502429 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:20.502463 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:20.504657 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.505294 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.505300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.505305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.505453 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.505854 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.505858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.505860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.506303 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:20.506430 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.507078 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.507461 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1911,"thread_start_us":114,"threads_started":1}
I20260927 02:47:20.509029 16416 ts_tablet_manager.cc:1397] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.009s
I20260927 02:47:20.511003 16524 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260927 02:47:20.511226 16524 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:20.511735 16524 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260927 02:47:20.511503 16416 raft_consensus.cc:348] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:47:20.512008 16416 raft_consensus.cc:729] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a983a30eda4b2584783d6af62cefd9, State: Initialized, Role: FOLLOWER
I20260927 02:47:20.512485 16416 consensus_queue.cc:260] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } }
I20260927 02:47:20.513441 16416 ts_tablet_manager.cc:1428] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:47:20.515008 16336 ts_manager.cc:194] Registered new tserver with Master: 51a983a30eda4b2584783d6af62cefd9 (127.0.98.1:39659)
I20260927 02:47:20.518858 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58485
I20260927 02:47:20.521113 16524 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
Sep 27 02:47:20 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:20.528422 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.518263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:39782)
0927 02:47:20.518908 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:20.518964 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:20.519095 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:20.519530 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:20.519535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:20.519942 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:20.520063 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.520072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.521401 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.521404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.522017 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.522026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.522604 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.522619 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.522726 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.523259 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:20.523273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:20.524750 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:20.526615 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.526637 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.526643 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.526871 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.527172 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.527177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.527179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.527264 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.527719 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:20.527724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:20.527848 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 02:47:20.528055 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 02:47:20.528250 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":107,"threads_started":1}
I20260927 02:47:20.528436 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.518302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.0.98.1:39659)
0927 02:47:20.518673 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.518685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.519199 (+ 514us) server_negotiation.cc:407] Connection header received
0927 02:47:20.519250 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.519254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.519392 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.519529 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:20.520204 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.521279 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.522160 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.522481 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.522601 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.524891 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:20.524899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:20.524908 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:20.524928 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:20.526479 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.526986 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.526999 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.527000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.527074 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.527366 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.527370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.527371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.527595 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:20.527743 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.528148 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.528295 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":86,"threads_started":1}
I20260927 02:47:20.529304 16458 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4d7f145d0f4a58a7563844b350f1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39782
I20260927 02:47:20.530092 16533 tablet_replica.cc:330] stopping tablet replica
I20260927 02:47:20.530385 16533 raft_consensus.cc:2229] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:20.530618 16533 raft_consensus.cc:2258] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:20.531634 16533 ts_tablet_manager.cc:1905] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:20.535962 16533 ts_tablet_manager.cc:1918] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:20.536069 16533 log.cc:1196] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal/wals/6c4d7f145d0f4a58a7563844b350f1b1
I20260927 02:47:20.536345 16533 ts_tablet_manager.cc:1939] T 6c4d7f145d0f4a58a7563844b350f1b1 P 51a983a30eda4b2584783d6af62cefd9: Deleting consensus metadata
I20260927 02:47:20.538195 16323 catalog_manager.cc:4841] TS 51a983a30eda4b2584783d6af62cefd9 (127.0.98.1:39659): tablet 6c4d7f145d0f4a58a7563844b350f1b1 (table test-table [id=a90c2fdaf2994fb184a9caf3ff63503a]) successfully deleted
W20260927 02:47:20.613943 16528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:20.614199 16528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:20.614269 16528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:20.618275 16528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:20.618327 16528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:20.618415 16528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:20.623534 16528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41417
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:20.624603 16528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:20.625646 16528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:20.635134 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:20.635141 16539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:20.635141 16538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:20.635365 16528 server_base.cc:1034] running on GCE node
I20260927 02:47:20.635946 16528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:20.636626 16528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:20.637822 16528 hybrid_clock.cc:648] HybridClock initialized: now 1790477240637814 us; error 40 us; skew 500 ppm
Sep 27 02:47:20 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:20.640588 16528 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:20.641556 16528 webserver.cc:460] Webserver started at http://127.0.98.2:41417/ using document root <none> and password file <none>
I20260927 02:47:20.642119 16528 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:20.642231 16528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:20.646384 16528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:47:20.648564 16548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.649816 16528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:47:20.649926 16528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "2ee4d138614a4400b2f5681b40850016"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "566d8ca95092e3b31a735f5557ce6d55"
server_key_iv: "5c4afd9cbdea4eeaadf90e8df9682d66"
server_key_version: "encryptionkey@0"
I20260927 02:47:20.650445 16528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:20.678239 16528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:20.681523 16528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:20.681727 16528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:20.682332 16528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:20.683135 16528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:20.683183 16528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.683221 16528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:20.683238 16528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.692337 16528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45947
I20260927 02:47:20.692363 16661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45947 every 8 connection(s)
I20260927 02:47:20.693290 16528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:20.699302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 16528
I20260927 02:47:20.699805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40247
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41909
--webserver_port=39075
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:47:20 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:20.707512 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.695063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35539 (local address 127.0.98.62:41909)
0927 02:47:20.695227 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.695232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.696463 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 02:47:20.697338 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.697342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.697398 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.697502 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:20.698624 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.699173 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.700564 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.700802 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.700915 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.703392 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:20.703398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:20.703401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:20.703436 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:20.705255 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.705790 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.705797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.705801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.705898 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.706274 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.706280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.706285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.706551 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:20.706682 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.707118 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.707263 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:47:20.708184 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.695608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:35539)
0927 02:47:20.696313 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:20.696353 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:20.697138 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:20.697669 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:20.697675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:20.698015 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:20.698477 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.698490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.699310 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.699314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.700437 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.700452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.700950 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.700954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.701128 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.701719 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:20.701734 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:20.703218 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:20.705390 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.705394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.705404 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.705685 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.706020 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.706023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.706026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.706168 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.706667 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:20.706672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:20.706883 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 02:47:20.707044 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 02:47:20.707285 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":100,"threads_started":1}
I20260927 02:47:20.709422 16662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260927 02:47:20.709681 16662 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:20.710160 16662 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260927 02:47:20.711181 16336 ts_manager.cc:194] Registered new tserver with Master: 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
I20260927 02:47:20.711723 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35539
W20260927 02:47:20.811967 16665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:20.812212 16665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:20.812253 16665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:20.816222 16665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:20.816272 16665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:20.816349 16665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:20.821435 16665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39075
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:20.822413 16665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:20.823375 16665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:20.833101 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:20.833114 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:20.833283 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:20.833263 16665 server_base.cc:1034] running on GCE node
I20260927 02:47:20.833802 16665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:20.834443 16665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:20.835632 16665 hybrid_clock.cc:648] HybridClock initialized: now 1790477240835619 us; error 36 us; skew 500 ppm
Sep 27 02:47:20 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:20.838490 16665 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:20.839329 16665 webserver.cc:460] Webserver started at http://127.0.98.3:39075/ using document root <none> and password file <none>
I20260927 02:47:20.839915 16665 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:20.840029 16665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:20.844167 16665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:20.846241 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.847632 16665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:20.847769 16665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "ec6216da7fa042aaaead9e00c56092ce"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "c4ce21621329b2295b1e11cef80e795d"
server_key_iv: "81605e4c482f8c1d233b057c1c036a32"
server_key_version: "encryptionkey@0"
I20260927 02:47:20.848446 16665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:20.865902 16665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:20.869046 16665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:20.869204 16665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:20.869735 16665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:20.870514 16665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:20.870563 16665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.870605 16665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:20.870628 16665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.880299 16665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40247
I20260927 02:47:20.880332 16793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40247 every 8 connection(s)
I20260927 02:47:20.881297 16665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:20.886308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 16665
Sep 27 02:47:20 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:20.895407 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.887095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.98.62:41909)
0927 02:47:20.887484 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.887489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.887505 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:47:20.887657 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.887661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.887713 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.887805 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:20.888550 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.889082 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.890108 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.890332 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.890440 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.891820 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:20.891823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:20.891826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:20.891868 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:20.893755 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.894179 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.894184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.894186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.894270 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.894556 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.894561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.894563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.894743 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:20.895017 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.895099 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.895221 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"spinlock_wait_cycles":768,"thread_start_us":103,"threads_started":1}
I20260927 02:47:20.895722 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.883089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52733 (local address 127.0.98.62:41909)
0927 02:47:20.883236 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.883240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.884346 (+ 1106us) server_negotiation.cc:407] Connection header received
0927 02:47:20.885180 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.885183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.885228 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.885305 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:20.886697 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.887409 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.888690 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.888925 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.889068 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.891683 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:20.891686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:20.891688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:20.891710 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:20.893566 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.894147 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.894151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.894153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.894238 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.894621 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.894625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.894627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.894814 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:20.894932 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.895457 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.895573 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":2304}
I20260927 02:47:20.896829 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.883476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:52733)
0927 02:47:20.884197 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:20.884236 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:20.884982 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:20.885516 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:20.885523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:20.885931 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:20.886521 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.886539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.887564 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.887568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.888539 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.888549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.889104 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.889114 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.889303 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.889981 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:20.889998 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:20.891492 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:20.893712 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.893719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.893734 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.894012 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.894344 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.894347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.894350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.894496 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.894934 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:20.894942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:20.895180 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 02:47:20.895393 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 02:47:20.895719 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":112,"threads_started":1}
I20260927 02:47:20.897969 16794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260927 02:47:20.898165 16794 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:20.898576 16794 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260927 02:47:20.899542 16335 ts_manager.cc:194] Registered new tserver with Master: ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247)
I20260927 02:47:20.900177 16335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52733
I20260927 02:47:20.909708 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:47:20.914124 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.910363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.0.98.62:41909)
0927 02:47:20.910530 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.910534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.910724 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:47:20.910849 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.910851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.910949 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.911045 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:20.911323 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.911835 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.912904 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.913162 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.913245 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.913588 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:20.913720 (+ 132us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 02:47:20.913946 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56264: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 02:47:20.914206 16798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56264: token expired
W20260927 02:47:20.914534 16264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:41909)
I20260927 02:47:20.922029 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.914933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.0.98.62:41909)
0927 02:47:20.915126 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.915131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.915149 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:47:20.915188 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.915191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.915236 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.915342 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:20.916060 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.916597 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.917404 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.917627 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.917719 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.919085 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:20.919088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:20.919090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:20.919110 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:20.920466 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.920864 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.920875 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.920876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.920955 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.921224 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.921229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.921230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.921390 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:20.921493 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.921696 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.921838 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 02:47:20.924520 16335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:20.927719 16335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:20.935446 16458 tablet_service.cc:1467] Processing CreateTablet for tablet 6688e4a58e5744da891462edb482797d (DEFAULT_TABLE table=test-table [id=d2fc943778064807a520a5461802d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:20.936084 16458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6688e4a58e5744da891462edb482797d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:20.938786 16416 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Bootstrap starting.
I20260927 02:47:20.939777 16416 tablet_bootstrap.cc:654] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:20.939810 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.934518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45947 (local address 127.0.0.1:50468)
0927 02:47:20.934784 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:20.934798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:20.934896 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:20.935811 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:20.935814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:20.935831 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:20.935967 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.935982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.936998 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.937002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.938201 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.938212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.939315 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.939319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.939415 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.939484 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 02:47:20.939540 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:47:20.939612 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20260927 02:47:20.940526 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.934758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.0.98.2:45947)
0927 02:47:20.935356 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.935365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.935384 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:47:20.935445 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.935449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.935663 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.935839 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:20.936077 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.936846 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.938344 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.939168 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.939999 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.940186 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.940264 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.940361 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":108,"threads_started":1}
I20260927 02:47:20.941422 16416 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: No bootstrap required, opened a new log
I20260927 02:47:20.941525 16416 ts_tablet_manager.cc:1397] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:47:20.941947 16416 raft_consensus.cc:348] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:20.942070 16416 raft_consensus.cc:374] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:20.942108 16416 raft_consensus.cc:729] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a983a30eda4b2584783d6af62cefd9, State: Initialized, Role: FOLLOWER
I20260927 02:47:20.942229 16416 consensus_queue.cc:260] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:20.942481 16416 ts_tablet_manager.cc:1428] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:20.942806 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.934138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40247 (local address 127.0.0.1:59150)
0927 02:47:20.934297 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:20.934312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:20.934392 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:20.939102 (+ 4710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:20.939105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:20.939123 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:20.939218 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.939224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.940407 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.940414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.941496 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.941509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.942461 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.942465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.942588 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.942611 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:47:20.942663 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:47:20.942720 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 02:47:20.943487 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.934311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.0.98.3:40247)
0927 02:47:20.934857 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.934871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.934895 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:47:20.934972 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.934978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.938948 (+ 3970us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.939127 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:20.939304 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.940272 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.941660 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.942284 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.943025 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.943185 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.943235 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.943340 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":175,"threads_started":1}
I20260927 02:47:20.943776 16596 tablet_service.cc:1467] Processing CreateTablet for tablet 6688e4a58e5744da891462edb482797d (DEFAULT_TABLE table=test-table [id=d2fc943778064807a520a5461802d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:20.945004 16596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6688e4a58e5744da891462edb482797d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:20.946352 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 6688e4a58e5744da891462edb482797d (DEFAULT_TABLE table=test-table [id=d2fc943778064807a520a5461802d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:20.947386 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6688e4a58e5744da891462edb482797d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:20.953132 16807 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: Bootstrap starting.
I20260927 02:47:20.954798 16808 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: Bootstrap starting.
I20260927 02:47:20.956112 16807 tablet_bootstrap.cc:654] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:20.956619 16808 tablet_bootstrap.cc:654] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:20.956956 16807 log.cc:824] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:20.957607 16808 log.cc:824] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:20.959081 16807 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: No bootstrap required, opened a new log
I20260927 02:47:20.959285 16807 ts_tablet_manager.cc:1397] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:47:20.959487 16808 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: No bootstrap required, opened a new log
I20260927 02:47:20.959669 16808 ts_tablet_manager.cc:1397] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:47:20.962538 16808 raft_consensus.cc:348] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:20.962560 16807 raft_consensus.cc:348] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:20.962777 16808 raft_consensus.cc:374] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:20.962773 16807 raft_consensus.cc:374] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:20.962845 16808 raft_consensus.cc:729] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6216da7fa042aaaead9e00c56092ce, State: Initialized, Role: FOLLOWER
I20260927 02:47:20.962848 16807 raft_consensus.cc:729] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee4d138614a4400b2f5681b40850016, State: Initialized, Role: FOLLOWER
I20260927 02:47:20.963264 16808 consensus_queue.cc:260] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:20.963274 16807 consensus_queue.cc:260] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:20.963842 16794 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
I20260927 02:47:20.963920 16662 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
I20260927 02:47:20.964213 16808 ts_tablet_manager.cc:1428] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:47:20.964280 16807 ts_tablet_manager.cc:1428] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 02:47:20.990315 16525 tablet.cc:2369] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:47:21.042297 16530 raft_consensus.cc:482] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:21.042526 16530 raft_consensus.cc:504] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:21.043915 16530 leader_election.cc:288] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
I20260927 02:47:21.048066 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.044306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33293 (local address 127.0.98.2:45947)
0927 02:47:21.044456 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.044461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.044656 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:47:21.044827 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.044831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.044883 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.044942 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.045330 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.045970 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.047064 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.047472 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.047588 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.047652 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.047840 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.047936 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:47:21.048075 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.044269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51039 (local address 127.0.98.3:40247)
0927 02:47:21.044395 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.044400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.044628 (+ 228us) server_negotiation.cc:407] Connection header received
0927 02:47:21.044859 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.044862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.044906 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.044998 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.045312 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.045946 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.047063 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.047487 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.047609 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.047667 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.047907 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.047978 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:47:21.048094 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.044157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45947 (local address 127.0.98.1:33293)
0927 02:47:21.044564 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.044576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.044701 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.045030 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.045033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.045075 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.045201 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.045207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.046106 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.046109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.046930 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.046945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.047593 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.047597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.047682 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.047737 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.047800 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.047914 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":48,"spinlock_wait_cycles":7936,"thread_start_us":61,"threads_started":1}
I20260927 02:47:21.048108 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.044151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40247 (local address 127.0.98.1:51039)
0927 02:47:21.044494 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.044516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.044684 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.045025 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.045028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.045072 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.045204 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.045208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.046054 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.046061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.046933 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.046942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.047640 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.047644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.047744 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.047761 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.047857 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.047948 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":30,"spinlock_wait_cycles":31744,"thread_start_us":70,"threads_started":1}
I20260927 02:47:21.049108 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6688e4a58e5744da891462edb482797d" candidate_uuid: "51a983a30eda4b2584783d6af62cefd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6216da7fa042aaaead9e00c56092ce" is_pre_election: true
I20260927 02:47:21.049372 16748 raft_consensus.cc:2454] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a983a30eda4b2584783d6af62cefd9 in term 0.
I20260927 02:47:21.049842 16410 leader_election.cc:302] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a983a30eda4b2584783d6af62cefd9, ec6216da7fa042aaaead9e00c56092ce; no voters:
I20260927 02:47:21.049882 16616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6688e4a58e5744da891462edb482797d" candidate_uuid: "51a983a30eda4b2584783d6af62cefd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee4d138614a4400b2f5681b40850016" is_pre_election: true
I20260927 02:47:21.050139 16616 raft_consensus.cc:2454] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a983a30eda4b2584783d6af62cefd9 in term 0.
I20260927 02:47:21.050199 16530 raft_consensus.cc:2781] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:21.050267 16530 raft_consensus.cc:482] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:21.050360 16530 raft_consensus.cc:3037] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.051432 16530 raft_consensus.cc:504] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:21.051788 16530 leader_election.cc:288] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 1 election: Requested vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
I20260927 02:47:21.052138 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6688e4a58e5744da891462edb482797d" candidate_uuid: "51a983a30eda4b2584783d6af62cefd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6216da7fa042aaaead9e00c56092ce"
I20260927 02:47:21.052177 16616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6688e4a58e5744da891462edb482797d" candidate_uuid: "51a983a30eda4b2584783d6af62cefd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee4d138614a4400b2f5681b40850016"
I20260927 02:47:21.052265 16748 raft_consensus.cc:3037] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.052295 16616 raft_consensus.cc:3037] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.053329 16748 raft_consensus.cc:2454] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a983a30eda4b2584783d6af62cefd9 in term 1.
I20260927 02:47:21.053553 16616 raft_consensus.cc:2454] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a983a30eda4b2584783d6af62cefd9 in term 1.
I20260927 02:47:21.053628 16410 leader_election.cc:302] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a983a30eda4b2584783d6af62cefd9, ec6216da7fa042aaaead9e00c56092ce; no voters:
I20260927 02:47:21.053817 16530 raft_consensus.cc:2781] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:21.054037 16530 raft_consensus.cc:686] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 LEADER]: Becoming Leader. State: Replica: 51a983a30eda4b2584783d6af62cefd9, State: Running, Role: LEADER
I20260927 02:47:21.054299 16530 consensus_queue.cc:237] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:21.057471 16335 catalog_manager.cc:5495] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a983a30eda4b2584783d6af62cefd9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "51a983a30eda4b2584783d6af62cefd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } health_report { overall_health: HEALTHY } } }
I20260927 02:47:21.092973 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.089234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.0.98.1:39659)
0927 02:47:21.089570 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.089577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.089588 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:47:21.089657 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.089659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.089710 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.089849 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:21.090107 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.090623 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.091506 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.091728 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.091842 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.092139 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:21.092564 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:21.092697 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.092760 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.092813 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":126,"threads_started":1}
I20260927 02:47:21.097069 16616 raft_consensus.cc:1261] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 1 FOLLOWER]: Refusing update from remote peer 51a983a30eda4b2584783d6af62cefd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:21.097069 16748 raft_consensus.cc:1261] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 1 FOLLOWER]: Refusing update from remote peer 51a983a30eda4b2584783d6af62cefd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:21.097754 16815 consensus_queue.cc:1035] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:21.097970 16530 consensus_queue.cc:1035] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:21.104604 16818 mvcc.cc:205] Tried to move back new op lower bound from 7333794779526680576 to 7333794779361099776. Current Snapshot: MvccSnapshot[applied={T|T < 7333794779526680576}]
I20260927 02:47:21.105989 16820 mvcc.cc:205] Tried to move back new op lower bound from 7333794779526680576 to 7333794779361099776. Current Snapshot: MvccSnapshot[applied={T|T < 7333794779526680576}]
I20260927 02:47:21.112792 16334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:21.112923 16334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:21.115078 16334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Sending DeleteTablet for 3 replicas of tablet 6688e4a58e5744da891462edb482797d
I20260927 02:47:21.115728 16728 tablet_service.cc:1514] Processing DeleteTablet for tablet 6688e4a58e5744da891462edb482797d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59150
I20260927 02:47:21.115842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 16374
I20260927 02:47:21.115964 16596 tablet_service.cc:1514] Processing DeleteTablet for tablet 6688e4a58e5744da891462edb482797d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50468
I20260927 02:47:21.116721 16828 tablet_replica.cc:330] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: stopping tablet replica
I20260927 02:47:21.116775 16827 tablet_replica.cc:330] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: stopping tablet replica
I20260927 02:47:21.117019 16828 raft_consensus.cc:2229] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:21.117026 16827 raft_consensus.cc:2229] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:21.117293 16828 raft_consensus.cc:2258] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:21.117331 16827 raft_consensus.cc:2258] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:21.118616 16828 ts_tablet_manager.cc:1905] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:21.120771 16828 ts_tablet_manager.cc:1918] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:21.120880 16828 log.cc:1196] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/6688e4a58e5744da891462edb482797d
I20260927 02:47:21.121237 16828 ts_tablet_manager.cc:1939] T 6688e4a58e5744da891462edb482797d P 2ee4d138614a4400b2f5681b40850016: Deleting consensus metadata
I20260927 02:47:21.121798 16827 ts_tablet_manager.cc:1905] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:21.122284 16323 catalog_manager.cc:4841] TS 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947): tablet 6688e4a58e5744da891462edb482797d (table test-table [id=d2fc943778064807a520a5461802d1e5]) successfully deleted
W20260927 02:47:21.123132 16323 connection.cc:535] client connection to 127.0.98.1:39659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 02:47:21.123304 16323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:47:21.123540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 16528
I20260927 02:47:21.124696 16827 ts_tablet_manager.cc:1918] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:21.124820 16827 log.cc:1196] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/6688e4a58e5744da891462edb482797d
I20260927 02:47:21.125178 16827 ts_tablet_manager.cc:1939] T 6688e4a58e5744da891462edb482797d P ec6216da7fa042aaaead9e00c56092ce: Deleting consensus metadata
I20260927 02:47:21.126371 16320 catalog_manager.cc:4841] TS ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247): tablet 6688e4a58e5744da891462edb482797d (table test-table [id=d2fc943778064807a520a5461802d1e5]) successfully deleted
I20260927 02:47:21.126750 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.126491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:39794)
0927 02:47:21.126626 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.126687 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:47:21.126977 16323 catalog_manager.cc:4584] TS 51a983a30eda4b2584783d6af62cefd9 (127.0.98.1:39659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6688e4a58e5744da891462edb482797d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
I20260927 02:47:21.130558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 16665
I20260927 02:47:21.137737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 16303
I20260927 02:47:41.144295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41909
--webserver_interface=127.0.98.62
--webserver_port=33735
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:41.252096 16830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.252334 16830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.252373 16830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.256637 16830 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:47:41.256691 16830 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:47:41.256708 16830 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:47:41.256722 16830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:41.256740 16830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.256757 16830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.256773 16830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:41.256793 16830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:41.261858 16830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:41.262801 16830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.263739 16830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.272336 16836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.272403 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.272396 16838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.272539 16830 server_base.cc:1034] running on GCE node
I20260927 02:47:41.273124 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.274145 16830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.275336 16830 hybrid_clock.cc:648] HybridClock initialized: now 1790477261275304 us; error 52 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.278048 16830 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:41.278841 16830 webserver.cc:460] Webserver started at http://127.0.98.62:33735/ using document root <none> and password file <none>
I20260927 02:47:41.279333 16830 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.279399 16830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.283084 16830 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:41.285270 16845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.286530 16830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:41.286666 16830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "05713974609043c69425adb1f2f2b824"
format_stamp: "Formatted at 2026-09-27 02:46:58 on dist-test-slave-gkrl"
server_key: "436fbab39fdaaaa1ed7dd21c49766560"
server_key_iv: "ad94c86b33f623be7345cd315760f527"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.287187 16830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.304651 16830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.307365 16830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.307569 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.314435 16830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41909
I20260927 02:47:41.314738 16897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41909 every 8 connection(s)
I20260927 02:47:41.315497 16830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:41.319622 16898 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:47:41.320928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 16830
I20260927 02:47:41.321324 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Bootstrap starting.
I20260927 02:47:41.321430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39659
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41909
--webserver_port=33085
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:47:41.338475 16898 log.cc:824] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:41.349720 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:47:41.350100 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Bootstrap complete.
I20260927 02:47:41.354035 16898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:47:41.354389 16898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05713974609043c69425adb1f2f2b824, State: Initialized, Role: FOLLOWER
I20260927 02:47:41.354899 16898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:47:41.355016 16898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:41.355098 16898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:41.355214 16898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:47:41.357062 16898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:47:41.357383 16898 leader_election.cc:302] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05713974609043c69425adb1f2f2b824; no voters:
I20260927 02:47:41.357697 16898 leader_election.cc:288] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:47:41.357777 16902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:47:41.357995 16902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [term 3 LEADER]: Becoming Leader. State: Replica: 05713974609043c69425adb1f2f2b824, State: Running, Role: LEADER
I20260927 02:47:41.358269 16902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260927 02:47:41.358899 16898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:41.359122 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05713974609043c69425adb1f2f2b824. Latest consensus state: current_term: 3 leader_uuid: "05713974609043c69425adb1f2f2b824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } } }
I20260927 02:47:41.359436 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:41.359134 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "05713974609043c69425adb1f2f2b824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05713974609043c69425adb1f2f2b824" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } } }
I20260927 02:47:41.359768 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:41.360554 16907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:41.365728 16907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:41.366559 16907 catalog_manager.cc:1259] Loaded cluster ID: 66fc94ead515456eb16fd1eb51400168
I20260927 02:47:41.366614 16907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:41.368989 16907 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:41.369679 16907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Loaded TSK: 0
I20260927 02:47:41.369755 16907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05713974609043c69425adb1f2f2b824: Loaded TSK: 1
F20260927 02:47:41.370427 16907 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f68c51b19ad google::LogMessage::Fail() at ??:0
@ 0x7f68c51b488a google::LogMessage::SendToLog() at ??:0
@ 0x7f68c51b14de google::LogMessage::Flush() at ??:0
@ 0x7f68c51b2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f68c7e7518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f68c7e6df0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f68cfb167bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f68cfaf099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f68cfaf1178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f68cfb32df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f68cfb5182b std::function<>::operator()() at ??:0
@ 0x7f68cfaf0b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f68cfaf1f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f68cfaee579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f68cfb32640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f68c74033d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f68c7403c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f68c74053f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f68c73f4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f68c8a386db start_thread at ??:0
@ 0x7f68c3caf61f clone at ??:0
W20260927 02:47:41.458678 16900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.458895 16900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.458936 16900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.463286 16900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.463332 16900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.463389 16900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:41.469264 16900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33085
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:41.470218 16900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.471165 16900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.481957 16945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.481967 16948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.482167 16900 server_base.cc:1034] running on GCE node
W20260927 02:47:41.481971 16946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.482738 16900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.483386 16900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.484608 16900 hybrid_clock.cc:648] HybridClock initialized: now 1790477261484569 us; error 48 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.487432 16900 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:41.488370 16900 webserver.cc:460] Webserver started at http://127.0.98.1:33085/ using document root <none> and password file <none>
I20260927 02:47:41.488919 16900 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.488979 16900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.492887 16900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:41.495043 16955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.496465 16900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:47:41.496603 16900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "51a983a30eda4b2584783d6af62cefd9"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "79e529c3915be5780954676e7675d7bb"
server_key_iv: "729306ae2712850cc1f8e087bbb78a17"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.497143 16900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.520865 16900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.524288 16900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.524490 16900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.525086 16900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:41.526151 16962 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:47:41.531486 16900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:47:41.531577 16900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 02:47:41.531663 16900 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:47:41.533344 16900 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:47:41.533412 16900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:47:41.533701 16962 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Bootstrap starting.
I20260927 02:47:41.543864 16900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39659
I20260927 02:47:41.544139 17069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39659 every 8 connection(s)
I20260927 02:47:41.545042 16900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:41.548166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 16900
I20260927 02:47:41.548918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45947
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41909
--webserver_port=41417
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:47:41.556226 16962 log.cc:824] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:41.562883 16962 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:47:41.563248 16962 tablet_bootstrap.cc:492] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Bootstrap complete.
I20260927 02:47:41.564456 16962 ts_tablet_manager.cc:1397] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.008s
I20260927 02:47:41.566726 16962 raft_consensus.cc:348] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:41.567134 16962 raft_consensus.cc:729] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a983a30eda4b2584783d6af62cefd9, State: Initialized, Role: FOLLOWER
I20260927 02:47:41.567617 16962 consensus_queue.cc:260] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:41.568578 16962 ts_tablet_manager.cc:1428] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
W20260927 02:47:41.639032 17072 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:41.547004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:46271)
0927 02:47:41.547470 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.547522 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.548664 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.638229 (+ 89565us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":140,"threads_started":1}
W20260927 02:47:41.639670 17070 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41909 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41909: Client connection negotiation failed: client connection to 127.0.98.62:41909: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:47:41.640453 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.640188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:45923)
0927 02:47:41.640316 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.640399 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 02:47:41.641160 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.640953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:48085)
0927 02:47:41.641060 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.641112 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:47:41.641351 17070 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:47:41.680702 17073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.680956 17073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.681035 17073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.685014 17073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.685091 17073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.685179 17073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:41.690384 17073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41417
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:41.691388 17073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.692468 17073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.702150 17083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.702150 17081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.702190 17080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.702479 17073 server_base.cc:1034] running on GCE node
I20260927 02:47:41.702881 17073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.703493 17073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.704674 17073 hybrid_clock.cc:648] HybridClock initialized: now 1790477261704652 us; error 31 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.707387 17073 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:41.708300 17073 webserver.cc:460] Webserver started at http://127.0.98.2:41417/ using document root <none> and password file <none>
I20260927 02:47:41.708845 17073 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.708935 17073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.712574 17073 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:47:41.714524 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.715612 17073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:41.715739 17073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "2ee4d138614a4400b2f5681b40850016"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "566d8ca95092e3b31a735f5557ce6d55"
server_key_iv: "5c4afd9cbdea4eeaadf90e8df9682d66"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.716346 17073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.729851 17073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.733052 17073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.733251 17073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.733820 17073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:41.734642 17073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:41.734711 17073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.734782 17073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:41.734826 17073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.743762 17073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45947
I20260927 02:47:41.743788 17203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45947 every 8 connection(s)
I20260927 02:47:41.744953 17073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:41.746555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 17073
I20260927 02:47:41.746999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40247
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41909
--webserver_port=39075
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:47:41.748883 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.747286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:34983)
0927 02:47:41.748031 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.748163 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":134,"threads_started":1}
W20260927 02:47:41.749418 17204 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41909 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41909: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
I20260927 02:47:41.750023 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.749804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:58873)
0927 02:47:41.749913 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.749967 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 02:47:41.750619 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.750422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:39841)
0927 02:47:41.750520 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.750565 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:47:41.750773 17204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:47:41.856551 17207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.856822 17207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.856894 17207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.860875 17207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.860950 17207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.861056 17207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:41.866230 17207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39075
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:41.867262 17207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.868327 17207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.877766 17213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.877761 17215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.877861 17212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.878374 17207 server_base.cc:1034] running on GCE node
I20260927 02:47:41.878762 17207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.879451 17207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.880657 17207 hybrid_clock.cc:648] HybridClock initialized: now 1790477261880633 us; error 31 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-gkrl krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.883523 17207 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:41.884480 17207 webserver.cc:460] Webserver started at http://127.0.98.3:39075/ using document root <none> and password file <none>
I20260927 02:47:41.885068 17207 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.885162 17207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.888900 17207 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:47:41.890906 17222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.892143 17207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:47:41.892266 17207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "ec6216da7fa042aaaead9e00c56092ce"
format_stamp: "Formatted at 2026-09-27 02:46:59 on dist-test-slave-gkrl"
server_key: "c4ce21621329b2295b1e11cef80e795d"
server_key_iv: "81605e4c482f8c1d233b057c1c036a32"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.892823 17207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.931463 17207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.934995 17207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.935209 17207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.935814 17207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:41.936657 17207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:41.936738 17207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.936808 17207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:41.936860 17207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.945732 17207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40247
I20260927 02:47:41.945824 17335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40247 every 8 connection(s)
I20260927 02:47:41.946748 17207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:41.950619 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.948867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:53289)
0927 02:47:41.949323 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.949905 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
W20260927 02:47:41.951054 17336 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41909 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41909: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
I20260927 02:47:41.951656 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.951420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:43459)
0927 02:47:41.951541 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.951594 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 02:47:41.952239 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.952076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:50559)
0927 02:47:41.952156 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.952195 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:47:41.952410 17336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:47:41.953655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 17207
I20260927 02:47:42.642382 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.641925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:34495)
0927 02:47:42.642190 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:42.642267 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260927 02:47:42.751766 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.751324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:59423)
0927 02:47:42.751618 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:42.751673 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260927 02:47:42.850059 17342 raft_consensus.cc:482] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:42.850281 17342 raft_consensus.cc:504] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:42.851480 17342 leader_election.cc:288] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
Sep 27 02:47:42 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 02:47:42 dist-test-slave-gkrl krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 02:47:42.865290 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.851706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40247 (local address 127.0.98.1:40213)
0927 02:47:42.851921 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:42.851937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:42.852053 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:42.854109 (+ 2056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:42.854114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:42.854476 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:42.854939 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:42.854953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:42.856307 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:42.856309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:42.857350 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:42.857361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:42.858067 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:42.858071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:42.858305 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:42.859006 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:42.859018 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:42.861012 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:42.863166 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:42.863169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:42.863171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:42.863365 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:42.863699 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:42.863704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:42.863705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:42.863788 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:42.864374 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:42.864378 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:42.864574 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 02:47:42.864758 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 02:47:42.865100 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":112,"spinlock_wait_cycles":30592}
I20260927 02:47:42.865303 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.851992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40213 (local address 127.0.98.3:40247)
0927 02:47:42.852287 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:42.852303 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:42.852345 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:47:42.853244 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:42.853252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:42.853708 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:42.854041 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:42.855110 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:42.856153 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:42.857484 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:42.857946 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:42.858121 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:42.861196 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:42.861201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:42.861211 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:42.861264 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:42.863033 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:42.863481 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:42.863486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:42.863491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:42.863602 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:42.863907 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:42.863910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:42.863911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:42.864266 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:42.864403 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:42.864875 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 02:47:42.865123 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
I20260927 02:47:42.865347 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.852059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39127 (local address 127.0.98.2:45947)
0927 02:47:42.852429 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:42.852448 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:42.852478 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:47:42.853244 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:42.853253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:42.853708 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:42.854041 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:42.855133 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:42.856152 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:42.857497 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:42.857952 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:42.858128 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:42.860916 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:42.860921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:42.860934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:42.860960 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:42.862763 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:42.863293 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:42.863304 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:42.863308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:42.863419 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:42.863763 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:42.863767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:42.863769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:42.864217 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:42.864360 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:42.864871 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 02:47:42.865162 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":61,"threads_started":1}
I20260927 02:47:42.865348 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.851756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45947 (local address 127.0.98.1:39127)
0927 02:47:42.852169 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:42.852184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:42.852284 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:42.854531 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:42.854533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:42.854693 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:42.854939 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:42.854954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:42.856299 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:42.856308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:42.857366 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:42.857378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:42.858096 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:42.858099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:42.858278 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:42.858964 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:42.858980 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:42.860667 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:42.862881 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:42.862886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:42.862901 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:42.863171 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:42.863523 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:42.863526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:42.863528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:42.863666 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:42.864335 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:42.864342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:42.864575 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 02:47:42.864758 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 02:47:42.865185 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":315,"spinlock_wait_cycles":2816,"thread_start_us":88,"threads_started":1}
W20260927 02:47:42.868615 16959 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
W20260927 02:47:42.869310 16956 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
I20260927 02:47:42.869382 16956 leader_election.cc:302] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a983a30eda4b2584783d6af62cefd9; no voters: 2ee4d138614a4400b2f5681b40850016, ec6216da7fa042aaaead9e00c56092ce
I20260927 02:47:42.869599 17342 raft_consensus.cc:2726] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:42.953270 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.952902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:48677)
0927 02:47:42.953130 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:42.953181 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260927 02:47:43.643597 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:43.643159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:44069)
0927 02:47:43.643430 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:43.643489 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260927 02:47:43.752882 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:43.752503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:40837)
0927 02:47:43.752744 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:43.752796 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260927 02:47:43.954628 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:43.954176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:57791)
0927 02:47:43.954452 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:43.954520 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
I20260927 02:47:44.644923 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:44.644524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:37497)
0927 02:47:44.644773 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:44.644829 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260927 02:47:44.754063 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:44.753595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:51247)
0927 02:47:44.753890 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:44.753941 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260927 02:47:44.778887 17352 raft_consensus.cc:482] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:44.779026 17352 raft_consensus.cc:504] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:44.779485 17352 leader_election.cc:288] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
W20260927 02:47:44.780321 16956 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
W20260927 02:47:44.780427 16959 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
I20260927 02:47:44.780520 16959 leader_election.cc:302] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a983a30eda4b2584783d6af62cefd9; no voters: 2ee4d138614a4400b2f5681b40850016, ec6216da7fa042aaaead9e00c56092ce
I20260927 02:47:44.780679 17352 raft_consensus.cc:2726] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:44.955801 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:44.955366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:50207)
0927 02:47:44.955612 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:44.955667 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260927 02:47:45.646251 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:45.645698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:58575)
0927 02:47:45.646051 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:45.646137 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260927 02:47:45.755362 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:45.754898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:41399)
0927 02:47:45.755185 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:45.755246 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
I20260927 02:47:45.957100 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:45.956677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:51105)
0927 02:47:45.956937 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:45.957000 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
I20260927 02:47:46.648005 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:46.647449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:51351)
0927 02:47:46.647764 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:46.647820 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":73,"threads_started":1}
I20260927 02:47:46.756544 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:46.756098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:35581)
0927 02:47:46.756376 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:46.756455 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260927 02:47:46.958292 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:46.957895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:48355)
0927 02:47:46.958147 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:46.958201 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260927 02:47:47.649209 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:47.648789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:33729)
0927 02:47:47.649056 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:47.649112 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260927 02:47:47.757601 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:47.757201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:48873)
0927 02:47:47.757439 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:47.757511 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260927 02:47:47.959517 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:47.959102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:42403)
0927 02:47:47.959367 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:47.959426 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260927 02:47:48.010056 17363 raft_consensus.cc:482] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:48.010180 17363 raft_consensus.cc:504] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:48.010579 17363 leader_election.cc:288] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
W20260927 02:47:48.011349 16959 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
W20260927 02:47:48.011658 16956 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
I20260927 02:47:48.011762 16956 leader_election.cc:302] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a983a30eda4b2584783d6af62cefd9; no voters: 2ee4d138614a4400b2f5681b40850016, ec6216da7fa042aaaead9e00c56092ce
I20260927 02:47:48.011957 17363 raft_consensus.cc:2726] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:48.650400 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:48.650023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:50499)
0927 02:47:48.650253 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:48.650305 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260927 02:47:48.758869 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:48.758403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:46333)
0927 02:47:48.758714 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:48.758771 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":73,"threads_started":1}
I20260927 02:47:48.961004 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:48.960484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:60563)
0927 02:47:48.960789 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:48.960851 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":75,"threads_started":1}
I20260927 02:47:49.651537 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:49.651099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:48559)
0927 02:47:49.651377 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:49.651442 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260927 02:47:49.760035 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:49.759599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:51891)
0927 02:47:49.759831 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:49.759944 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260927 02:47:49.962288 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:49.961842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:40787)
0927 02:47:49.962102 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:49.962153 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
I20260927 02:47:50.652904 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:50.652388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:52525)
0927 02:47:50.652713 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:50.652782 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260927 02:47:50.761087 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:50.760701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:60247)
0927 02:47:50.760950 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:50.761003 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260927 02:47:50.963429 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:50.963037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:33617)
0927 02:47:50.963283 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:50.963339 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260927 02:47:51.443800 17373 raft_consensus.cc:482] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:51.443979 17373 raft_consensus.cc:504] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:51.444471 17373 leader_election.cc:288] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
W20260927 02:47:51.445230 16959 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
W20260927 02:47:51.445322 16956 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
I20260927 02:47:51.445379 16956 leader_election.cc:302] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a983a30eda4b2584783d6af62cefd9; no voters: 2ee4d138614a4400b2f5681b40850016, ec6216da7fa042aaaead9e00c56092ce
I20260927 02:47:51.445508 17373 raft_consensus.cc:2726] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:51.654211 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:51.653798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:49763)
0927 02:47:51.654056 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:51.654120 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260927 02:47:51.762243 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:51.761851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:51699)
0927 02:47:51.762100 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:51.762154 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260927 02:47:51.964725 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:51.964331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:42517)
0927 02:47:51.964582 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:51.964636 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260927 02:47:52.655380 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:52.654953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:47561)
0927 02:47:52.655231 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:52.655289 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260927 02:47:52.763377 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:52.762978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:53063)
0927 02:47:52.763234 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:52.763288 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260927 02:47:52.965879 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:52.965488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:55737)
0927 02:47:52.965735 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:52.965785 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
I20260927 02:47:53.642786 17380 raft_consensus.cc:482] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:53.642947 17380 raft_consensus.cc:504] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6216da7fa042aaaead9e00c56092ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40247 } } peers { permanent_uuid: "2ee4d138614a4400b2f5681b40850016" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45947 } } peers { permanent_uuid: "51a983a30eda4b2584783d6af62cefd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260927 02:47:53.643486 17380 leader_election.cc:288] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247), 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947)
W20260927 02:47:53.644239 16956 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec6216da7fa042aaaead9e00c56092ce (127.0.98.3:40247): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
W20260927 02:47:53.644363 16959 leader_election.cc:341] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee4d138614a4400b2f5681b40850016 (127.0.98.2:45947): Not found: Tablet not found: 6688e4a58e5744da891462edb482797d
I20260927 02:47:53.644472 16959 leader_election.cc:302] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a983a30eda4b2584783d6af62cefd9; no voters: 2ee4d138614a4400b2f5681b40850016, ec6216da7fa042aaaead9e00c56092ce
I20260927 02:47:53.644622 17380 raft_consensus.cc:2726] T 6688e4a58e5744da891462edb482797d P 51a983a30eda4b2584783d6af62cefd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:53.656545 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:53.656187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:37093)
0927 02:47:53.656409 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:53.656463 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":70,"threads_started":1}
I20260927 02:47:53.764568 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:53.764163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:34431)
0927 02:47:53.764423 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:53.764478 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260927 02:47:53.967087 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:53.966616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:45093)
0927 02:47:53.966935 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:53.966992 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":88,"threads_started":1}
I20260927 02:47:54.657837 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:54.657369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:47063)
0927 02:47:54.657658 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:54.657742 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20260927 02:47:54.765775 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:54.765353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:41641)
0927 02:47:54.765624 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:54.765681 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260927 02:47:54.968470 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:54.968030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:56915)
0927 02:47:54.968309 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:54.968368 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260927 02:47:55.659173 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:55.658705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:48969)
0927 02:47:55.659018 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:55.659074 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":105,"threads_started":1}
I20260927 02:47:55.767033 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:55.766605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:33461)
0927 02:47:55.766886 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:55.766941 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260927 02:47:55.969663 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:55.969213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:59519)
0927 02:47:55.969501 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:55.969563 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20260927 02:47:56.660655 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:56.660223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:37047)
0927 02:47:56.660493 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:56.660561 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260927 02:47:56.768332 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:56.767929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:40549)
0927 02:47:56.768183 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:56.768240 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260927 02:47:56.964710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 16900
I20260927 02:47:56.970798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 17073
I20260927 02:47:56.971094 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:56.970591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:33619)
0927 02:47:56.970898 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:56.970967 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":70,"threads_started":1}
I20260927 02:47:56.976619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 17207
2026-09-27T02:47:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59336 ms)
[----------] 30 tests from SecurityITest (245054 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:57 dist-test-slave-gkrl krb5kdc[17395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:57 dist-test-slave-gkrl krb5kdc[17395](info): set up 2 sockets
Sep 27 02:47:57 dist-test-slave-gkrl krb5kdc[17395](info): commencing operation
krb5kdc: starting...
W20260927 02:47:58.020876 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:58 dist-test-slave-gkrl krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:58.144944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:58.253124 17408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:58.253381 17408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:58.253422 17408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:58.257321 17408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:58.257378 17408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:58.257398 17408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:58.257414 17408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:58.257431 17408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:58.262548 17408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:58.263504 17408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:58.264472 17408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:58.273187 17413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:58.273231 17416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:58.273253 17414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:58.273490 17408 server_base.cc:1034] running on GCE node
I20260927 02:47:58.273952 17408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:58.275039 17408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:58.276278 17408 hybrid_clock.cc:648] HybridClock initialized: now 1790477278276248 us; error 41 us; skew 500 ppm
Sep 27 02:47:58 dist-test-slave-gkrl krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:58.283351 17408 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:58.284598 17408 webserver.cc:460] Webserver started at http://127.0.98.62:38087/ using document root <none> and password file <none>
I20260927 02:47:58.285400 17408 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:58.285543 17408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:58.285835 17408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:58.288601 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "c5f273d28a604d8d9045baad37796c2c"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "d55146310ce32f3e6142074ce57e7e4b"
server_key_iv: "91f04e98a545f640ad015958a2d8282b"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.289228 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5f273d28a604d8d9045baad37796c2c"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "d55146310ce32f3e6142074ce57e7e4b"
server_key_iv: "91f04e98a545f640ad015958a2d8282b"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.294299 17408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 02:47:58.296792 17423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:58.298404 17408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:47:58.298533 17408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "c5f273d28a604d8d9045baad37796c2c"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "d55146310ce32f3e6142074ce57e7e4b"
server_key_iv: "91f04e98a545f640ad015958a2d8282b"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.298668 17408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:58.315243 17408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:58.319520 17408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:58.319733 17408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:58.329931 17408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40825
I20260927 02:47:58.330063 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40825 every 8 connection(s)
I20260927 02:47:58.331328 17408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:58.334266 17476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:58.340219 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c: Bootstrap starting.
I20260927 02:47:58.341154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 17408
I20260927 02:47:58.341292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:58.341684 392 external_mini_cluster.cc:1428] Setting key ff7b6c1b26c905144b682d66cf545461
I20260927 02:47:58.343288 17476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:58.344177 17476 log.cc:824] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:58.346344 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c: No bootstrap required, opened a new log
I20260927 02:47:58.350510 17476 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f273d28a604d8d9045baad37796c2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } }
I20260927 02:47:58.350718 17476 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:58.350800 17476 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f273d28a604d8d9045baad37796c2c, State: Initialized, Role: FOLLOWER
I20260927 02:47:58.351203 17476 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f273d28a604d8d9045baad37796c2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } }
Sep 27 02:47:58 dist-test-slave-gkrl krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:58.351327 17476 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:58.351421 17476 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:58.351521 17476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:58.352761 17476 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f273d28a604d8d9045baad37796c2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } }
I20260927 02:47:58.353129 17476 leader_election.cc:302] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5f273d28a604d8d9045baad37796c2c; no voters:
I20260927 02:47:58.353538 17476 leader_election.cc:288] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:58.353644 17481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:58.353926 17481 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [term 1 LEADER]: Becoming Leader. State: Replica: c5f273d28a604d8d9045baad37796c2c, State: Running, Role: LEADER
I20260927 02:47:58.354288 17481 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f273d28a604d8d9045baad37796c2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } }
I20260927 02:47:58.354817 17476 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:58.356045 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5f273d28a604d8d9045baad37796c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f273d28a604d8d9045baad37796c2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } }
I20260927 02:47:58.356477 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:58.356364 17482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5f273d28a604d8d9045baad37796c2c. Latest consensus state: current_term: 1 leader_uuid: "c5f273d28a604d8d9045baad37796c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f273d28a604d8d9045baad37796c2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } }
I20260927 02:47:58.356812 17482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:58.358223 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:58.342817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.0.98.62:40825)
0927 02:47:58.343255 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:58.343274 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:58.343317 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:47:58.344212 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:58.344237 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:58.344663 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:58.344980 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:58.345693 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.347134 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.348127 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.348938 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.349190 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:58.352036 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:58.352043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:58.352059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:58.352089 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:58.354197 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:58.354761 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:58.354767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:58.354780 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:58.354925 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:58.355214 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:58.355217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:58.355219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:58.355800 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:58.356041 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:58.356953 (+ 912us) server_negotiation.cc:299] Negotiation successful
0927 02:47:58.357274 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
W20260927 02:47:58.360287 17496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:47:58.360361 17496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:47:58.360456 17495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:58.361218 17495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:58.366554 17495 catalog_manager.cc:1347] Generated new cluster ID: 0f0bc5067efc4a71a21da40896dbcec9
I20260927 02:47:58.366616 17495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:58.377584 17495 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:58.378746 17495 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:58.387158 17495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c: Generated new TSK 0
I20260927 02:47:58.387784 17495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:58.421149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40825
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:58.530453 17502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:58.530680 17502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:58.530727 17502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:58.534665 17502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:58.534724 17502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:58.534816 17502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:58.539918 17502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:58.540866 17502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:58.541834 17502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:58.550906 17508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:58.550966 17507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:58.550976 17510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:58.551165 17502 server_base.cc:1034] running on GCE node
I20260927 02:47:58.551574 17502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:58.552248 17502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:58.553408 17502 hybrid_clock.cc:648] HybridClock initialized: now 1790477278553393 us; error 27 us; skew 500 ppm
Sep 27 02:47:58 dist-test-slave-gkrl krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:58.556284 17502 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:58.557192 17502 webserver.cc:460] Webserver started at http://127.0.98.1:33037/ using document root <none> and password file <none>
I20260927 02:47:58.557731 17502 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:58.557791 17502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:58.558008 17502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:58.559990 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "5130cee98ae44b9ca629f8b580d09422"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "23161f2407ea5463ce087d45f67abfd5"
server_key_iv: "7cf86a30236d95f6db36c4cc0e4d850c"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.560527 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5130cee98ae44b9ca629f8b580d09422"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "23161f2407ea5463ce087d45f67abfd5"
server_key_iv: "7cf86a30236d95f6db36c4cc0e4d850c"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.563959 17502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 02:47:58.566200 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:58.567365 17502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:47:58.567489 17502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "5130cee98ae44b9ca629f8b580d09422"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "23161f2407ea5463ce087d45f67abfd5"
server_key_iv: "7cf86a30236d95f6db36c4cc0e4d850c"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.567592 17502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:58.592444 17502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:58.595193 17502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:58.595393 17502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:58.596004 17502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:58.596782 17502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:58.596849 17502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:58.596918 17502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:58.596971 17502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:58.605947 17502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42987
I20260927 02:47:58.605975 17630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42987 every 8 connection(s)
I20260927 02:47:58.606786 17502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:58.607398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 17502
I20260927 02:47:58.607529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:58.607839 392 external_mini_cluster.cc:1428] Setting key 093c350e2dc07e49e422576fdc5095ff
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 02:47:58 dist-test-slave-gkrl krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:47:58.622462 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:58.609324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36647 (local address 127.0.98.62:40825)
0927 02:47:58.609471 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:58.609477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:58.610425 (+ 948us) server_negotiation.cc:407] Connection header received
0927 02:47:58.611454 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:58.611459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:58.611526 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:58.611642 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:58.613085 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.613729 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.615600 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.615838 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.615997 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:58.618772 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:58.618775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:58.618777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:58.618801 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:58.620285 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:58.620752 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:58.620755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:58.620756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:58.620858 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:58.621236 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:58.621243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:58.621247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:58.621498 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:58.621580 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:58.622244 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 02:47:58.622340 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:47:58.623624 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:58.609734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40825 (local address 127.0.98.1:36647)
0927 02:47:58.610229 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:58.610294 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:58.611252 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:58.611781 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:58.611787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:58.612216 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:58.612916 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:58.612939 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.613872 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.613877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:58.615468 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:58.615483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.616060 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.616067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:58.616308 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:58.616934 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:58.616957 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:58.618615 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:58.620391 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:58.620396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:58.620406 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:58.620664 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:58.620987 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:58.620992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:58.620995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:58.621131 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:58.621607 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:58.621614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:58.621993 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 02:47:58.622205 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 02:47:58.622562 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":146,"threads_started":1}
I20260927 02:47:58.625020 17631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40825
I20260927 02:47:58.625273 17631 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:58.625799 17631 heartbeater.cc:511] Master 127.0.98.62:40825 requested a full tablet report, sending...
I20260927 02:47:58.627297 17440 ts_manager.cc:194] Registered new tserver with Master: 5130cee98ae44b9ca629f8b580d09422 (127.0.98.1:42987)
I20260927 02:47:58.628436 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:58.639384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40825
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:58.747536 17636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:58.747797 17636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:58.747866 17636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:58.751812 17636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:58.751866 17636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:58.751986 17636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:58.757315 17636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:58.758314 17636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:58.759306 17636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:58.768464 17644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:58.768517 17642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:58.768479 17641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:58.768667 17636 server_base.cc:1034] running on GCE node
I20260927 02:47:58.769125 17636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:58.769726 17636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:58.770891 17636 hybrid_clock.cc:648] HybridClock initialized: now 1790477278770873 us; error 34 us; skew 500 ppm
Sep 27 02:47:58 dist-test-slave-gkrl krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:58.773790 17636 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:58.774552 17636 webserver.cc:460] Webserver started at http://127.0.98.2:45229/ using document root <none> and password file <none>
I20260927 02:47:58.775100 17636 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:58.775161 17636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:58.775314 17636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:58.777413 17636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "d8e4bad8ee0b48a7aed7a8576cb17135"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "c8234dd13bee8b6d7bee8f27e897b266"
server_key_iv: "bab19e27158555b19eb5c08ae708d870"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.777904 17636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d8e4bad8ee0b48a7aed7a8576cb17135"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "c8234dd13bee8b6d7bee8f27e897b266"
server_key_iv: "bab19e27158555b19eb5c08ae708d870"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.781366 17636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:47:58.783576 17651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:58.784930 17636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:58.785040 17636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "d8e4bad8ee0b48a7aed7a8576cb17135"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "c8234dd13bee8b6d7bee8f27e897b266"
server_key_iv: "bab19e27158555b19eb5c08ae708d870"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.785144 17636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:58.808158 17636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:58.811529 17636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:58.811743 17636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:58.812378 17636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:58.813185 17636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:58.813231 17636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:58.813300 17636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:58.813340 17636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:58.822484 17636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35041
I20260927 02:47:58.822502 17764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35041 every 8 connection(s)
I20260927 02:47:58.823369 17636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:58.825525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 17636
I20260927 02:47:58.825675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:58.826042 392 external_mini_cluster.cc:1428] Setting key e20967fb11c4a14751c4a50dc2bd984c
Sep 27 02:47:58 dist-test-slave-gkrl krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:47:58.837649 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:58.825133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51115 (local address 127.0.98.62:40825)
0927 02:47:58.825307 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:58.825312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:58.826285 (+ 973us) server_negotiation.cc:407] Connection header received
0927 02:47:58.827306 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:58.827311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:58.827355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:58.827448 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:58.829024 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.829595 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.830768 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.831034 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.831152 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:58.833674 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:58.833681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:58.833685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:58.833715 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:58.835386 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:58.836093 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:58.836097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:58.836099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:58.836188 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:58.836554 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:58.836556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:58.836557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:58.836732 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:58.836848 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:58.837409 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 02:47:58.837526 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:47:58.838526 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:58.825534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40825 (local address 127.0.98.2:51115)
0927 02:47:58.826101 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:58.826161 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:58.827101 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:58.827628 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:58.827638 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:58.828162 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:58.828868 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:58.828890 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.829739 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.829743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:58.830635 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:58.830645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:58.831195 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:58.831200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:58.831368 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:58.832044 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:58.832061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:58.833503 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:58.835531 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:58.835536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:58.835547 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:58.835991 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:58.836312 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:58.836314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:58.836316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:58.836453 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:58.836855 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:58.836865 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:58.837139 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 02:47:58.837302 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 02:47:58.837568 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":155,"threads_started":1}
I20260927 02:47:58.839730 17765 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40825
I20260927 02:47:58.840075 17765 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:58.840626 17765 heartbeater.cc:511] Master 127.0.98.62:40825 requested a full tablet report, sending...
I20260927 02:47:58.841665 17440 ts_manager.cc:194] Registered new tserver with Master: d8e4bad8ee0b48a7aed7a8576cb17135 (127.0.98.2:35041)
I20260927 02:47:58.842242 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:58.855232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40825
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:58.966501 17770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:58.966732 17770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:58.966773 17770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:58.970731 17770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:58.970783 17770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:58.970839 17770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:58.975958 17770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:58.976884 17770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:58.977787 17770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:58.987340 17776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:58.987526 17770 server_base.cc:1034] running on GCE node
W20260927 02:47:58.987377 17778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:58.987360 17775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:58.988124 17770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:58.988767 17770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:58.989967 17770 hybrid_clock.cc:648] HybridClock initialized: now 1790477278989952 us; error 34 us; skew 500 ppm
Sep 27 02:47:58 dist-test-slave-gkrl krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:58.992707 17770 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:58.993590 17770 webserver.cc:460] Webserver started at http://127.0.98.3:36683/ using document root <none> and password file <none>
I20260927 02:47:58.994134 17770 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:58.994226 17770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:58.994453 17770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:58.996449 17770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7089468489546a1b725e00d7eff5d8d"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "f7f4fed8b80f0b1056453a99f8b5cece"
server_key_iv: "5d8bfa5f59dec79d7a91fedb39b6ec57"
server_key_version: "encryptionkey@0"
I20260927 02:47:58.996994 17770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7089468489546a1b725e00d7eff5d8d"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "f7f4fed8b80f0b1056453a99f8b5cece"
server_key_iv: "5d8bfa5f59dec79d7a91fedb39b6ec57"
server_key_version: "encryptionkey@0"
I20260927 02:47:59.000452 17770 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:47:59.003015 17785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:59.004370 17770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:59.004468 17770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "d7089468489546a1b725e00d7eff5d8d"
format_stamp: "Formatted at 2026-09-27 02:47:58 on dist-test-slave-gkrl"
server_key: "f7f4fed8b80f0b1056453a99f8b5cece"
server_key_iv: "5d8bfa5f59dec79d7a91fedb39b6ec57"
server_key_version: "encryptionkey@0"
I20260927 02:47:59.004549 17770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:59.035826 17770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:59.039148 17770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:59.039309 17770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:59.039841 17770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:59.040656 17770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:59.040702 17770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:59.040741 17770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:59.040757 17770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:59.049867 17770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34055
I20260927 02:47:59.049891 17898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34055 every 8 connection(s)
I20260927 02:47:59.050722 17770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:59.051666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 17770
I20260927 02:47:59.051787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:59.052084 392 external_mini_cluster.cc:1428] Setting key ddded4f29225213a7c6f10b3d29fe4e4
Sep 27 02:47:59 dist-test-slave-gkrl krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:59.064433 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.052737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50713 (local address 127.0.98.62:40825)
0927 02:47:59.052900 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.052904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.053870 (+ 966us) server_negotiation.cc:407] Connection header received
0927 02:47:59.054925 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.054927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.054971 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.055039 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:59.056427 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.057021 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.058183 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.058403 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.058495 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.060958 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:59.060961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:59.060962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:59.060986 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:59.062430 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:59.062951 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:59.062954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:59.062955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:59.063026 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:59.063378 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:59.063380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:59.063381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:59.063544 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:59.063622 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.064215 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.064316 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:47:59.065208 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.053143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40825 (local address 127.0.98.3:50713)
0927 02:47:59.053678 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.053765 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:59.054742 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:59.055336 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:59.055344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:59.055796 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:59.056290 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.056305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.057149 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.057152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.058043 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.058052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.058536 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.058539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.058678 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.059295 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:59.059311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:59.060806 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:59.062551 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:59.062555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:59.062566 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:59.062833 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:59.063130 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:59.063133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:59.063134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:59.063294 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:59.063661 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:59.063666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:59.063999 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 02:47:59.064163 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 02:47:59.064405 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260927 02:47:59.066493 17899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40825
I20260927 02:47:59.066725 17899 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:59.067198 17899 heartbeater.cc:511] Master 127.0.98.62:40825 requested a full tablet report, sending...
I20260927 02:47:59.068218 17439 ts_manager.cc:194] Registered new tserver with Master: d7089468489546a1b725e00d7eff5d8d (127.0.98.3:34055)
I20260927 02:47:59.068797 17439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50713
I20260927 02:47:59.078001 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:47:59.087199 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.079849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.0.98.62:40825)
0927 02:47:59.080009 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.080014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.080153 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:47:59.080309 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.080312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.080362 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.080447 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:59.081138 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.081646 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.082488 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.082715 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.082825 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.084073 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:59.084076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:59.084079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:59.084103 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:59.085416 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:59.085919 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:59.085925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:59.085929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:59.086014 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:59.086308 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:59.086313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:59.086315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:59.086537 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:59.086672 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.086914 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.087019 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 02:47:59.090389 17439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:59.093434 17439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:59.108593 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.102622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.98.2:35041)
0927 02:47:59.103954 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.103966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.103988 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:47:59.104062 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.104066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.104199 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.104344 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:59.104529 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.105243 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.107124 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.107807 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.107987 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.108139 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.108335 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.108442 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":128,"threads_started":1}
I20260927 02:47:59.108582 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.102427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35041 (local address 127.0.0.1:34956)
0927 02:47:59.102733 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.102747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:59.102841 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:59.104324 (+ 1483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:59.104327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:59.104344 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:59.104433 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.104439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.105376 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.105382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.106957 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.106966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.107985 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.107990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.108101 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.108141 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:47:59.108300 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 02:47:59.108385 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":156,"thread_start_us":90,"threads_started":1}
I20260927 02:47:59.108825 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.102522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.0.98.3:34055)
0927 02:47:59.103129 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.103142 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.103171 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:47:59.103230 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.103234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.103456 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.103626 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:59.104064 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.105293 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.107044 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.107815 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.107974 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.108125 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.108421 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.108521 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":130,"threads_started":1}
I20260927 02:47:59.108925 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.101884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.0.98.1:42987)
0927 02:47:59.103038 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.103053 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.103074 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:47:59.103187 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.103192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.103411 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.103863 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:59.104093 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.105302 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.106972 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.107794 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.108473 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.108600 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.108645 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.108751 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":129,"threads_started":1}
I20260927 02:47:59.109125 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.101701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42987 (local address 127.0.0.1:59208)
0927 02:47:59.102181 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.102233 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:59.102369 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:59.103613 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:59.103620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:59.103657 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:59.103944 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.103953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.105493 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.105497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.106846 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.106855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.108063 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.108066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.108162 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.108185 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:47:59.108234 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:47:59.108366 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":233,"spinlock_wait_cycles":6400,"thread_start_us":66,"threads_started":1}
I20260927 02:47:59.109493 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.101789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34055 (local address 127.0.0.1:39710)
0927 02:47:59.102684 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.102699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:59.102812 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:59.103613 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:59.103620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:59.103659 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:59.103905 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.103914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.105477 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.105484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.106873 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.106885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.108179 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.108182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.108266 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.108284 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:47:59.109164 (+ 880us) client_negotiation.cc:240] Negotiation successful
0927 02:47:59.109313 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":98,"spinlock_wait_cycles":7680,"thread_start_us":86,"threads_started":1}
I20260927 02:47:59.112093 17833 tablet_service.cc:1467] Processing CreateTablet for tablet b966297c1b004367beaf9e45de4258b5 (DEFAULT_TABLE table=test-table [id=abcae2f9d2964c21a42fc71da697fdb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:59.112241 17565 tablet_service.cc:1467] Processing CreateTablet for tablet b966297c1b004367beaf9e45de4258b5 (DEFAULT_TABLE table=test-table [id=abcae2f9d2964c21a42fc71da697fdb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:59.112241 17699 tablet_service.cc:1467] Processing CreateTablet for tablet b966297c1b004367beaf9e45de4258b5 (DEFAULT_TABLE table=test-table [id=abcae2f9d2964c21a42fc71da697fdb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:59.113106 17833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b966297c1b004367beaf9e45de4258b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:59.113219 17699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b966297c1b004367beaf9e45de4258b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:59.113219 17565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b966297c1b004367beaf9e45de4258b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:59.119168 17916 tablet_bootstrap.cc:492] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Bootstrap starting.
I20260927 02:47:59.120787 17917 tablet_bootstrap.cc:492] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422: Bootstrap starting.
I20260927 02:47:59.121325 17916 tablet_bootstrap.cc:654] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:59.122320 17916 log.cc:824] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:59.122483 17917 tablet_bootstrap.cc:654] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:59.123440 17917 log.cc:824] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:59.124291 17918 tablet_bootstrap.cc:492] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: Bootstrap starting.
I20260927 02:47:59.124500 17916 tablet_bootstrap.cc:492] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: No bootstrap required, opened a new log
I20260927 02:47:59.124696 17916 ts_tablet_manager.cc:1397] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:47:59.125963 17918 tablet_bootstrap.cc:654] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:59.127101 17918 log.cc:824] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:59.127827 17916 raft_consensus.cc:348] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.128120 17916 raft_consensus.cc:374] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:59.128181 17916 raft_consensus.cc:729] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7089468489546a1b725e00d7eff5d8d, State: Initialized, Role: FOLLOWER
I20260927 02:47:59.128659 17917 tablet_bootstrap.cc:492] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422: No bootstrap required, opened a new log
I20260927 02:47:59.128643 17916 consensus_queue.cc:260] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.128850 17917 ts_tablet_manager.cc:1397] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 02:47:59.128980 17918 tablet_bootstrap.cc:492] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: No bootstrap required, opened a new log
I20260927 02:47:59.129141 17918 ts_tablet_manager.cc:1397] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:47:59.129549 17916 ts_tablet_manager.cc:1428] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:47:59.129926 17899 heartbeater.cc:503] Master 127.0.98.62:40825 was elected leader, sending a full tablet report...
I20260927 02:47:59.132647 17918 raft_consensus.cc:348] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.132633 17917 raft_consensus.cc:348] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.132893 17917 raft_consensus.cc:374] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:59.132893 17918 raft_consensus.cc:374] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:59.132944 17917 raft_consensus.cc:729] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5130cee98ae44b9ca629f8b580d09422, State: Initialized, Role: FOLLOWER
I20260927 02:47:59.132943 17918 raft_consensus.cc:729] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e4bad8ee0b48a7aed7a8576cb17135, State: Initialized, Role: FOLLOWER
I20260927 02:47:59.133378 17918 consensus_queue.cc:260] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.133378 17917 consensus_queue.cc:260] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.133977 17631 heartbeater.cc:503] Master 127.0.98.62:40825 was elected leader, sending a full tablet report...
I20260927 02:47:59.134364 17918 ts_tablet_manager.cc:1428] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:47:59.134660 17765 heartbeater.cc:503] Master 127.0.98.62:40825 was elected leader, sending a full tablet report...
I20260927 02:47:59.135185 17917 ts_tablet_manager.cc:1428] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 02:47:59.301666 17900 tablet.cc:2369] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:47:59.324487 17766 tablet.cc:2369] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:47:59.358119 17632 tablet.cc:2369] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:47:59.406780 17922 raft_consensus.cc:482] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:59.407052 17922 raft_consensus.cc:504] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.408327 17922 leader_election.cc:288] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5130cee98ae44b9ca629f8b580d09422 (127.0.98.1:42987), d8e4bad8ee0b48a7aed7a8576cb17135 (127.0.98.2:35041)
I20260927 02:47:59.412456 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.408648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42987 (local address 127.0.98.3:58453)
0927 02:47:59.408821 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.408839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:59.408929 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:59.409235 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:59.409238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:59.409259 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:59.409413 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.409422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.410321 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.410324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.411291 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.411300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.411964 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.411967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.412078 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.412136 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 02:47:59.412210 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:47:59.412266 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":71}
I20260927 02:47:59.412487 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.408755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58453 (local address 127.0.98.1:42987)
0927 02:47:59.408932 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.408937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.408957 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:47:59.409023 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.409026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.409077 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.409174 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:59.409543 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.410200 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.411449 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.411840 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.411950 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.412025 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.412258 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.412341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:47:59.412501 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.408923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35041 (local address 127.0.98.3:58485)
0927 02:47:59.409275 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.409288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:59.409379 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:59.409626 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:59.409628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:59.409646 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:59.409759 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.409765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.410545 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.410548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.411330 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.411338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.412086 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.412089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.412180 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.412226 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 02:47:59.412304 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:47:59.412366 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":2,"thread_start_us":134,"threads_started":1}
I20260927 02:47:59.412540 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.409062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58485 (local address 127.0.98.2:35041)
0927 02:47:59.409196 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.409201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.409387 (+ 186us) server_negotiation.cc:407] Connection header received
0927 02:47:59.409448 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.409454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.409515 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.409585 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:59.409895 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.410434 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.411478 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.411954 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.412055 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.412120 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.412339 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.412405 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:47:59.413097 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b966297c1b004367beaf9e45de4258b5" candidate_uuid: "d7089468489546a1b725e00d7eff5d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" is_pre_election: true
I20260927 02:47:59.413184 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b966297c1b004367beaf9e45de4258b5" candidate_uuid: "d7089468489546a1b725e00d7eff5d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5130cee98ae44b9ca629f8b580d09422" is_pre_election: true
I20260927 02:47:59.413406 17719 raft_consensus.cc:2454] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7089468489546a1b725e00d7eff5d8d in term 0.
I20260927 02:47:59.413545 17585 raft_consensus.cc:2454] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7089468489546a1b725e00d7eff5d8d in term 0.
I20260927 02:47:59.413918 17788 leader_election.cc:302] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7089468489546a1b725e00d7eff5d8d, d8e4bad8ee0b48a7aed7a8576cb17135; no voters:
I20260927 02:47:59.414170 17922 raft_consensus.cc:2781] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:59.414258 17922 raft_consensus.cc:482] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:59.414301 17922 raft_consensus.cc:3037] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:59.415304 17922 raft_consensus.cc:504] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.415619 17922 leader_election.cc:288] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [CANDIDATE]: Term 1 election: Requested vote from peers 5130cee98ae44b9ca629f8b580d09422 (127.0.98.1:42987), d8e4bad8ee0b48a7aed7a8576cb17135 (127.0.98.2:35041)
I20260927 02:47:59.416021 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b966297c1b004367beaf9e45de4258b5" candidate_uuid: "d7089468489546a1b725e00d7eff5d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5130cee98ae44b9ca629f8b580d09422"
I20260927 02:47:59.416057 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b966297c1b004367beaf9e45de4258b5" candidate_uuid: "d7089468489546a1b725e00d7eff5d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135"
I20260927 02:47:59.416153 17719 raft_consensus.cc:3037] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:59.416153 17585 raft_consensus.cc:3037] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:59.417392 17719 raft_consensus.cc:2454] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7089468489546a1b725e00d7eff5d8d in term 1.
I20260927 02:47:59.417392 17585 raft_consensus.cc:2454] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7089468489546a1b725e00d7eff5d8d in term 1.
I20260927 02:47:59.417676 17788 leader_election.cc:302] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7089468489546a1b725e00d7eff5d8d, d8e4bad8ee0b48a7aed7a8576cb17135; no voters:
I20260927 02:47:59.417832 17922 raft_consensus.cc:2781] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:59.418057 17922 raft_consensus.cc:686] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 1 LEADER]: Becoming Leader. State: Replica: d7089468489546a1b725e00d7eff5d8d, State: Running, Role: LEADER
I20260927 02:47:59.418414 17922 consensus_queue.cc:237] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } }
I20260927 02:47:59.421424 17440 catalog_manager.cc:5495] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d reported cstate change: term changed from 0 to 1, leader changed from <none> to d7089468489546a1b725e00d7eff5d8d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d7089468489546a1b725e00d7eff5d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7089468489546a1b725e00d7eff5d8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260927 02:47:59.468696 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.465483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.0.98.3:34055)
0927 02:47:59.465611 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.465615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.465628 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:47:59.465701 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.465703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.465759 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.465848 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:59.466061 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.466562 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.467348 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.467581 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.467665 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.467955 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:59.468361 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:59.468467 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.468522 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.468572 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:47:59.474555 17585 raft_consensus.cc:1261] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [term 1 FOLLOWER]: Refusing update from remote peer d7089468489546a1b725e00d7eff5d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:59.474587 17719 raft_consensus.cc:1261] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 1 FOLLOWER]: Refusing update from remote peer d7089468489546a1b725e00d7eff5d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:59.475279 17926 consensus_queue.cc:1035] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8e4bad8ee0b48a7aed7a8576cb17135" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:59.475462 17922 consensus_queue.cc:1035] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5130cee98ae44b9ca629f8b580d09422" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:59.482928 17929 mvcc.cc:205] Tried to move back new op lower bound from 7333794936720871424 to 7333794936500580352. Current Snapshot: MvccSnapshot[applied={T|T < 7333794936720871424}]
I20260927 02:47:59.492321 17440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:59.492496 17440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:59.494812 17440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5f273d28a604d8d9045baad37796c2c: Sending DeleteTablet for 3 replicas of tablet b966297c1b004367beaf9e45de4258b5
I20260927 02:47:59.495600 17833 tablet_service.cc:1514] Processing DeleteTablet for tablet b966297c1b004367beaf9e45de4258b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39710
I20260927 02:47:59.495657 17565 tablet_service.cc:1514] Processing DeleteTablet for tablet b966297c1b004367beaf9e45de4258b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59208
I20260927 02:47:59.496096 17939 tablet_replica.cc:330] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: stopping tablet replica
I20260927 02:47:59.496284 17699 tablet_service.cc:1514] Processing DeleteTablet for tablet b966297c1b004367beaf9e45de4258b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34956
I20260927 02:47:59.496384 17939 raft_consensus.cc:2229] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:59.496754 17939 raft_consensus.cc:2258] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:59.496765 17941 tablet_replica.cc:330] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: stopping tablet replica
I20260927 02:47:59.497059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 17502
I20260927 02:47:59.497153 17940 tablet_replica.cc:330] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422: stopping tablet replica
I20260927 02:47:59.497447 17940 raft_consensus.cc:2229] T b966297c1b004367beaf9e45de4258b5 P 5130cee98ae44b9ca629f8b580d09422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:59.498557 17939 ts_tablet_manager.cc:1905] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:59.501379 17939 ts_tablet_manager.cc:1918] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:59.501487 17939 log.cc:1196] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/b966297c1b004367beaf9e45de4258b5
I20260927 02:47:59.501812 17939 ts_tablet_manager.cc:1939] T b966297c1b004367beaf9e45de4258b5 P d7089468489546a1b725e00d7eff5d8d: Deleting consensus metadata
I20260927 02:47:59.503001 17425 catalog_manager.cc:4841] TS d7089468489546a1b725e00d7eff5d8d (127.0.98.3:34055): tablet b966297c1b004367beaf9e45de4258b5 (table test-table [id=abcae2f9d2964c21a42fc71da697fdb0]) successfully deleted
I20260927 02:47:59.503037 17941 raft_consensus.cc:2229] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:59.503346 17941 raft_consensus.cc:2258] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:59.504719 17941 ts_tablet_manager.cc:1905] T b966297c1b004367beaf9e45de4258b5 P d8e4bad8ee0b48a7aed7a8576cb17135: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:47:59.505403 17426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42987 (error 108)
I20260927 02:47:59.505930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 17636
I20260927 02:47:59.511557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 17770
I20260927 02:47:59.511796 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.511494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42987 (local address 127.0.0.1:59210)
0927 02:47:59.511630 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.511706 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 02:47:59.512171 17426 catalog_manager.cc:4584] TS 5130cee98ae44b9ca629f8b580d09422 (127.0.98.1:42987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b966297c1b004367beaf9e45de4258b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42987: connect: Connection refused (error 111)
I20260927 02:47:59.513109 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.512862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35041 (local address 127.0.0.1:34964)
0927 02:47:59.512973 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.513032 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 02:47:59.516839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 17408
2026-09-27T02:47:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2539 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T02:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:59Z Disabled control of system clock
I20260927 02:47:59.545898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36903
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:47:59.652357 17946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:59.652601 17946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:59.652638 17946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:59.656530 17946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:59.656585 17946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:59.656600 17946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:59.656617 17946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:59.656632 17946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:59.661300 17946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:59.662245 17946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:59.663514 17946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:59.671932 17952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:59.672005 17946 server_base.cc:1034] running on GCE node
W20260927 02:47:59.672010 17954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:59.672120 17951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:59.672751 17946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:59.673806 17946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:59.674997 17946 hybrid_clock.cc:648] HybridClock initialized: now 1790477279674975 us; error 30 us; skew 500 ppm
I20260927 02:47:59.676838 17946 webserver.cc:460] Webserver started at http://127.0.98.62:36531/ using document root <none> and password file <none>
I20260927 02:47:59.677425 17946 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:59.677518 17946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:59.677744 17946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:59.679692 17946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "f9e75bb2d1eb4cbf871020041ccd0089"
format_stamp: "Formatted at 2026-09-27 02:47:59 on dist-test-slave-gkrl"
server_key: "99267797a1ae8b113238e153064bc3bd"
server_key_iv: "da717f4bae9c04548dc077d5b7b948b2"
server_key_version: "encryptionkey@0"
I20260927 02:47:59.680303 17946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9e75bb2d1eb4cbf871020041ccd0089"
format_stamp: "Formatted at 2026-09-27 02:47:59 on dist-test-slave-gkrl"
server_key: "99267797a1ae8b113238e153064bc3bd"
server_key_iv: "da717f4bae9c04548dc077d5b7b948b2"
server_key_version: "encryptionkey@0"
I20260927 02:47:59.683753 17946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:59.686062 17960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:59.687218 17946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:59.687354 17946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "f9e75bb2d1eb4cbf871020041ccd0089"
format_stamp: "Formatted at 2026-09-27 02:47:59 on dist-test-slave-gkrl"
server_key: "99267797a1ae8b113238e153064bc3bd"
server_key_iv: "da717f4bae9c04548dc077d5b7b948b2"
server_key_version: "encryptionkey@0"
I20260927 02:47:59.687464 17946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:59.709966 17946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:59.710597 17946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:59.710784 17946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:59.717733 17946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36903
I20260927 02:47:59.717769 18012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36903 every 8 connection(s)
I20260927 02:47:59.718642 17946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:59.721326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 17946
I20260927 02:47:59.721451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:59.721753 392 external_mini_cluster.cc:1428] Setting key b30c5dbd8b84a13b1812cb792c61e997
I20260927 02:47:59.721839 18013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:59.728055 18013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089: Bootstrap starting.
I20260927 02:47:59.730201 18013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:59.730862 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.723150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.0.98.62:36903)
0927 02:47:59.723780 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.723793 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.723826 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:47:59.724570 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.724587 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.724965 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.725314 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:59.725732 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.726692 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.727587 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.728151 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.728333 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.729216 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:59.729221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:59.729233 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:59.729609 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:59.729663 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:59.729682 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:59.729794 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.729909 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.730160 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":137,"threads_started":1}
I20260927 02:47:59.730899 18013 log.cc:824] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:59.732966 18013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089: No bootstrap required, opened a new log
I20260927 02:47:59.736398 18013 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e75bb2d1eb4cbf871020041ccd0089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } }
I20260927 02:47:59.736670 18013 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:59.736730 18013 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e75bb2d1eb4cbf871020041ccd0089, State: Initialized, Role: FOLLOWER
I20260927 02:47:59.737183 18013 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e75bb2d1eb4cbf871020041ccd0089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } }
I20260927 02:47:59.737311 18013 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:59.737360 18013 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:59.737426 18013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:59.738211 18013 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e75bb2d1eb4cbf871020041ccd0089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } }
I20260927 02:47:59.738480 18013 leader_election.cc:302] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9e75bb2d1eb4cbf871020041ccd0089; no voters:
I20260927 02:47:59.738700 18013 leader_election.cc:288] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:59.738848 18018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:59.739096 18018 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [term 1 LEADER]: Becoming Leader. State: Replica: f9e75bb2d1eb4cbf871020041ccd0089, State: Running, Role: LEADER
I20260927 02:47:59.739406 18018 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e75bb2d1eb4cbf871020041ccd0089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } }
I20260927 02:47:59.739940 18013 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:59.741078 18019 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9e75bb2d1eb4cbf871020041ccd0089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e75bb2d1eb4cbf871020041ccd0089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } } }
I20260927 02:47:59.741079 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e75bb2d1eb4cbf871020041ccd0089. Latest consensus state: current_term: 1 leader_uuid: "f9e75bb2d1eb4cbf871020041ccd0089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e75bb2d1eb4cbf871020041ccd0089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36903 } } }
I20260927 02:47:59.741235 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:59.741235 18019 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:59.741693 18027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:59.744692 18027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:59.749612 18027 catalog_manager.cc:1347] Generated new cluster ID: b6cb61c829684c74a6da442a2c172059
I20260927 02:47:59.749671 18027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:59.760563 18027 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:59.761485 18027 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:59.770700 18027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089: Generated new TSK 0
I20260927 02:47:59.771492 18027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:47:59.781621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--builtin_ntp_servers=127.0.98.20:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:47:59.891039 18037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:59.891304 18037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:59.891369 18037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:59.895489 18037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:59.895569 18037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:59.895653 18037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:59.900369 18037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:47:59.901381 18037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:59.902714 18037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:59.912084 18043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:59.912072 18045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:59.912249 18037 server_base.cc:1034] running on GCE node
W20260927 02:47:59.912487 18042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:59.912976 18037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:59.913548 18037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:59.914721 18037 hybrid_clock.cc:648] HybridClock initialized: now 1790477279914689 us; error 42 us; skew 500 ppm
I20260927 02:47:59.916584 18037 webserver.cc:460] Webserver started at http://127.0.98.1:45457/ using document root <none> and password file <none>
I20260927 02:47:59.917197 18037 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:59.917261 18037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:59.917483 18037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:59.919457 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc2086076ff049b28e5ebf90c0d20fa6"
format_stamp: "Formatted at 2026-09-27 02:47:59 on dist-test-slave-gkrl"
server_key: "92ec6f9797d12ba33ffe857ec06e1c43"
server_key_iv: "2018bc7d3cb45b2319182dab960983ad"
server_key_version: "encryptionkey@0"
I20260927 02:47:59.920053 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc2086076ff049b28e5ebf90c0d20fa6"
format_stamp: "Formatted at 2026-09-27 02:47:59 on dist-test-slave-gkrl"
server_key: "92ec6f9797d12ba33ffe857ec06e1c43"
server_key_iv: "2018bc7d3cb45b2319182dab960983ad"
server_key_version: "encryptionkey@0"
I20260927 02:47:59.923506 18037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:47:59.925864 18051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:59.927457 18037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 02:47:59.927580 18037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "fc2086076ff049b28e5ebf90c0d20fa6"
format_stamp: "Formatted at 2026-09-27 02:47:59 on dist-test-slave-gkrl"
server_key: "92ec6f9797d12ba33ffe857ec06e1c43"
server_key_iv: "2018bc7d3cb45b2319182dab960983ad"
server_key_version: "encryptionkey@0"
I20260927 02:47:59.927691 18037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:59.968546 18037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:59.969274 18037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:59.969481 18037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:59.970079 18037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:59.971012 18037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:59.971091 18037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:59.971161 18037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:59.971204 18037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:59.979707 18037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41987
I20260927 02:47:59.979730 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41987 every 8 connection(s)
I20260927 02:47:59.980603 18037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:59.988111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 18037
I20260927 02:47:59.988263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:59.988569 392 external_mini_cluster.cc:1428] Setting key b8c645bdbdfb018915d4af54ea443669
I20260927 02:47:59.990002 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.982431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38083 (local address 127.0.98.62:36903)
0927 02:47:59.982588 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:59.982592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:59.983638 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 02:47:59.985085 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:59.985088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:59.985142 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:59.985248 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:59.986123 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.986656 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.987873 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.988120 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.988235 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.989513 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:59.989517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:59.989519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:59.989530 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:59.989548 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:59.989556 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:59.989631 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:59.989792 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 02:47:59.989867 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:47:59.990594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--builtin_ntp_servers=127.0.98.20:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:47:59.990871 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.982863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36903 (local address 127.0.98.1:38083)
0927 02:47:59.983497 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.983537 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:59.984878 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:59.985427 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:59.985434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:59.985517 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:59.985983 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.985994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.986863 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.986867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.987749 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:59.987758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:59.988311 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:59.988316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:59.988515 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:59.989253 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:59.989254 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:59.989294 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:59.989354 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:59.989645 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:59.989654 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:59.989660 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:47:59.989792 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 02:47:59.990061 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260927 02:47:59.992882 18165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36903
I20260927 02:47:59.993114 18165 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:59.993616 18165 heartbeater.cc:511] Master 127.0.98.62:36903 requested a full tablet report, sending...
I20260927 02:47:59.995137 17977 ts_manager.cc:194] Registered new tserver with Master: fc2086076ff049b28e5ebf90c0d20fa6 (127.0.98.1:41987)
I20260927 02:47:59.996398 17977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38083
W20260927 02:48:00.105813 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:00.106077 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:00.106145 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:00.110168 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:00.110244 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:00.110333 18168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:00.114929 18168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:00.115916 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:00.117332 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:00.126546 18176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:00.126601 18174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:00.126638 18168 server_base.cc:1034] running on GCE node
W20260927 02:48:00.126546 18173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:00.127252 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:00.127949 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:00.129112 18168 hybrid_clock.cc:648] HybridClock initialized: now 1790477280129097 us; error 26 us; skew 500 ppm
I20260927 02:48:00.131024 18168 webserver.cc:460] Webserver started at http://127.0.98.2:41907/ using document root <none> and password file <none>
I20260927 02:48:00.131657 18168 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:00.131721 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:00.131959 18168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:00.133909 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "d12cb2439fcc4cd8b54d9fac8c474183"
format_stamp: "Formatted at 2026-09-27 02:48:00 on dist-test-slave-gkrl"
server_key: "6e9d91ddf40d6f54a99116b24e9dc7de"
server_key_iv: "6d0e2b9bb4ff12ec9ffb0db009c46b77"
server_key_version: "encryptionkey@0"
I20260927 02:48:00.134397 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d12cb2439fcc4cd8b54d9fac8c474183"
format_stamp: "Formatted at 2026-09-27 02:48:00 on dist-test-slave-gkrl"
server_key: "6e9d91ddf40d6f54a99116b24e9dc7de"
server_key_iv: "6d0e2b9bb4ff12ec9ffb0db009c46b77"
server_key_version: "encryptionkey@0"
I20260927 02:48:00.137871 18168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:48:00.140156 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:00.141573 18168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:48:00.141697 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "d12cb2439fcc4cd8b54d9fac8c474183"
format_stamp: "Formatted at 2026-09-27 02:48:00 on dist-test-slave-gkrl"
server_key: "6e9d91ddf40d6f54a99116b24e9dc7de"
server_key_iv: "6d0e2b9bb4ff12ec9ffb0db009c46b77"
server_key_version: "encryptionkey@0"
I20260927 02:48:00.141801 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:00.166986 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:00.167678 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:00.167932 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:00.168537 18168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:00.169458 18168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:00.169529 18168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:00.169598 18168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:00.169649 18168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:00.178965 18168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35043
I20260927 02:48:00.178992 18295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35043 every 8 connection(s)
I20260927 02:48:00.179857 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:00.188302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 18168
I20260927 02:48:00.188477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:00.188846 392 external_mini_cluster.cc:1428] Setting key 44b7bbf7de27457e83bb3c9864b7edf4
I20260927 02:48:00.189702 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.181807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42023 (local address 127.0.98.62:36903)
0927 02:48:00.182017 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.182022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.183056 (+ 1034us) server_negotiation.cc:407] Connection header received
0927 02:48:00.184570 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.184573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.184643 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.184729 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:00.185674 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.186199 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.187438 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.187648 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.187727 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.189112 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:00.189116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:00.189118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:00.189131 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:00.189152 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:00.189161 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:00.189236 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.189467 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.189546 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 02:48:00.190686 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.182221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36903 (local address 127.0.98.2:42023)
0927 02:48:00.182903 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.182945 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:00.184352 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:00.184914 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:00.184920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:00.185006 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:00.185507 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.185520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.186337 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.186341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.187309 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.187319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.187794 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.187797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.188042 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.188800 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:00.188803 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:00.188859 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:00.188955 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:00.189270 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:00.189281 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:00.189287 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:48:00.189461 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 02:48:00.189749 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260927 02:48:00.191064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--builtin_ntp_servers=127.0.98.20:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:48:00.191817 18296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36903
I20260927 02:48:00.192085 18296 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:00.192741 18296 heartbeater.cc:511] Master 127.0.98.62:36903 requested a full tablet report, sending...
I20260927 02:48:00.193882 17977 ts_manager.cc:194] Registered new tserver with Master: d12cb2439fcc4cd8b54d9fac8c474183 (127.0.98.2:35043)
I20260927 02:48:00.194460 17977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42023
W20260927 02:48:00.300199 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:00.300487 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:00.300570 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:00.304561 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:00.304641 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:00.304733 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:00.309360 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:00.310351 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:00.311643 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:00.321169 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:00.321228 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:00.321236 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:00.321413 18299 server_base.cc:1034] running on GCE node
I20260927 02:48:00.321799 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:00.322450 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:00.323648 18299 hybrid_clock.cc:648] HybridClock initialized: now 1790477280323623 us; error 42 us; skew 500 ppm
I20260927 02:48:00.325629 18299 webserver.cc:460] Webserver started at http://127.0.98.3:39517/ using document root <none> and password file <none>
I20260927 02:48:00.326212 18299 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:00.326279 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:00.326500 18299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:00.328756 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "55b37c388f4143c18b024a1b925b2393"
format_stamp: "Formatted at 2026-09-27 02:48:00 on dist-test-slave-gkrl"
server_key: "3a98bb04b03f70b172499a23ca04f263"
server_key_iv: "72201a9cd0fe8a4750b7f0e421decc96"
server_key_version: "encryptionkey@0"
I20260927 02:48:00.329319 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "55b37c388f4143c18b024a1b925b2393"
format_stamp: "Formatted at 2026-09-27 02:48:00 on dist-test-slave-gkrl"
server_key: "3a98bb04b03f70b172499a23ca04f263"
server_key_iv: "72201a9cd0fe8a4750b7f0e421decc96"
server_key_version: "encryptionkey@0"
I20260927 02:48:00.333010 18299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 02:48:00.335322 18313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:00.336644 18299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:00.336766 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "55b37c388f4143c18b024a1b925b2393"
format_stamp: "Formatted at 2026-09-27 02:48:00 on dist-test-slave-gkrl"
server_key: "3a98bb04b03f70b172499a23ca04f263"
server_key_iv: "72201a9cd0fe8a4750b7f0e421decc96"
server_key_version: "encryptionkey@0"
I20260927 02:48:00.336900 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:00.350361 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:00.350973 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:00.351159 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:00.351718 18299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:00.352638 18299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:00.352684 18299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:00.352744 18299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:00.352777 18299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:00.361928 18299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41101
I20260927 02:48:00.361982 18426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41101 every 8 connection(s)
I20260927 02:48:00.362797 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:00.367595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 18299
I20260927 02:48:00.367718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:00.368089 392 external_mini_cluster.cc:1428] Setting key 10b2912e9a155a9b5863b009e02ed849
I20260927 02:48:00.371959 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.364572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47229 (local address 127.0.98.62:36903)
0927 02:48:00.364718 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.364723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.365790 (+ 1067us) server_negotiation.cc:407] Connection header received
0927 02:48:00.367182 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.367186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.367251 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.367358 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:00.368406 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.368932 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.370073 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.370304 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.370415 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.371441 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:00.371452 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:00.371454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:00.371465 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:00.371483 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:00.371491 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:00.371570 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.371736 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.371796 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:48:00.372824 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.364968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36903 (local address 127.0.98.3:47229)
0927 02:48:00.365600 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.365640 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:00.366973 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:00.367528 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:00.367535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:00.367613 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:00.368260 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.368275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.369052 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.369056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.369946 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.369954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.370448 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.370451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.370615 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.371166 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:00.371168 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:00.371205 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:00.371269 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:00.371581 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:00.371596 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:00.371604 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 02:48:00.371735 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 02:48:00.372055 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260927 02:48:00.373862 18427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36903
I20260927 02:48:00.374094 18427 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:00.374565 18427 heartbeater.cc:511] Master 127.0.98.62:36903 requested a full tablet report, sending...
I20260927 02:48:00.375547 17977 ts_manager.cc:194] Registered new tserver with Master: 55b37c388f4143c18b024a1b925b2393 (127.0.98.3:41101)
I20260927 02:48:00.376189 17977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47229
I20260927 02:48:00.381701 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:48:00.387001 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.383533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.0.98.62:36903)
0927 02:48:00.383653 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.383657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.383871 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:48:00.384032 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.384034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.384070 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.384134 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:00.384392 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.384858 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.385640 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.385837 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.385927 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.386654 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:00.386658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:00.386659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:00.386670 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:00.386687 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:00.386694 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:00.386749 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.386854 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.386894 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:48:00.389828 17977 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:48:00.392827 17977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:00.407017 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.400831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41987 (local address 127.0.0.1:46124)
0927 02:48:00.401308 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.401366 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:00.401491 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:00.402690 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:00.402696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:00.402715 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:00.402831 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.402837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.404030 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.404035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.405357 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.405366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.406452 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.406459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.406562 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.406598 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 02:48:00.406730 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 02:48:00.406862 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":89,"thread_start_us":120,"threads_started":1}
I20260927 02:48:00.407039 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.401645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.0.98.1:41987)
0927 02:48:00.402241 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.402254 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.402277 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:48:00.402346 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.402351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.402545 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.402683 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:00.402984 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.403923 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.405486 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.406307 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.406437 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.406648 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.406760 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.406861 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":128,"threads_started":1}
I20260927 02:48:00.407063 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.401790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41101 (local address 127.0.0.1:47230)
0927 02:48:00.402757 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.402770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:00.402863 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:00.403313 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:00.403315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:00.403334 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:00.403430 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.403435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.404562 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.404568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.405756 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.405762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.406739 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.406743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.406824 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.406842 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:48:00.406905 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:48:00.406958 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":42,"thread_start_us":90,"threads_started":1}
I20260927 02:48:00.407579 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.401189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.98.2:35043)
0927 02:48:00.401787 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.401799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.401824 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:48:00.401906 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.401911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.402114 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.402608 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:00.402764 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.403951 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.405669 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.406439 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.406844 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.407008 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.407059 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.407163 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":130,"threads_started":1}
I20260927 02:48:00.407613 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.402099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.0.98.3:41101)
0927 02:48:00.402582 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.402596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.402854 (+ 258us) server_negotiation.cc:407] Connection header received
0927 02:48:00.402970 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.402975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.403151 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.403301 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:00.403549 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.404455 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.405863 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.406392 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.406505 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.406679 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.407358 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.407459 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":149,"threads_started":1}
I20260927 02:48:00.407603 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.400861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35043 (local address 127.0.0.1:57724)
0927 02:48:00.401721 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.401733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:00.401812 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:00.402295 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:00.402301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:00.402325 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:00.402453 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.402459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.404081 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.404083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.405545 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.405555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.406571 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.406574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.406660 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.406694 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:48:00.406747 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:48:00.407484 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":719,"thread_start_us":89,"threads_started":1}
I20260927 02:48:00.410131 18230 tablet_service.cc:1467] Processing CreateTablet for tablet 173adb7dae8a4f7ebde713ca2e72717c (DEFAULT_TABLE table=test-table [id=aa702693d86e45ae955e7f421985d294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:00.410215 18098 tablet_service.cc:1467] Processing CreateTablet for tablet 173adb7dae8a4f7ebde713ca2e72717c (DEFAULT_TABLE table=test-table [id=aa702693d86e45ae955e7f421985d294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:00.410233 18361 tablet_service.cc:1467] Processing CreateTablet for tablet 173adb7dae8a4f7ebde713ca2e72717c (DEFAULT_TABLE table=test-table [id=aa702693d86e45ae955e7f421985d294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:00.411175 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173adb7dae8a4f7ebde713ca2e72717c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:00.411176 18098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173adb7dae8a4f7ebde713ca2e72717c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:00.411175 18361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173adb7dae8a4f7ebde713ca2e72717c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:00.417441 18444 tablet_bootstrap.cc:492] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Bootstrap starting.
I20260927 02:48:00.418457 18445 tablet_bootstrap.cc:492] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Bootstrap starting.
I20260927 02:48:00.419804 18444 tablet_bootstrap.cc:654] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:00.420384 18445 tablet_bootstrap.cc:654] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:00.420601 18444 log.cc:824] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:00.421137 18445 log.cc:824] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:00.421204 18446 tablet_bootstrap.cc:492] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6: Bootstrap starting.
I20260927 02:48:00.422775 18444 tablet_bootstrap.cc:492] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: No bootstrap required, opened a new log
I20260927 02:48:00.422869 18445 tablet_bootstrap.cc:492] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: No bootstrap required, opened a new log
I20260927 02:48:00.423017 18444 ts_tablet_manager.cc:1397] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:48:00.423048 18445 ts_tablet_manager.cc:1397] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:48:00.423305 18446 tablet_bootstrap.cc:654] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:00.424149 18446 log.cc:824] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:00.426045 18446 tablet_bootstrap.cc:492] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6: No bootstrap required, opened a new log
I20260927 02:48:00.426221 18446 ts_tablet_manager.cc:1397] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:48:00.426756 18445 raft_consensus.cc:348] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.426977 18445 raft_consensus.cc:374] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:00.427028 18445 raft_consensus.cc:729] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b37c388f4143c18b024a1b925b2393, State: Initialized, Role: FOLLOWER
I20260927 02:48:00.427448 18445 consensus_queue.cc:260] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.427419 18444 raft_consensus.cc:348] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.427664 18444 raft_consensus.cc:374] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:00.427727 18444 raft_consensus.cc:729] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12cb2439fcc4cd8b54d9fac8c474183, State: Initialized, Role: FOLLOWER
I20260927 02:48:00.428337 18427 heartbeater.cc:503] Master 127.0.98.62:36903 was elected leader, sending a full tablet report...
I20260927 02:48:00.428457 18445 ts_tablet_manager.cc:1428] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:48:00.429406 18444 consensus_queue.cc:260] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.430374 18444 ts_tablet_manager.cc:1428] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 02:48:00.430748 18296 heartbeater.cc:503] Master 127.0.98.62:36903 was elected leader, sending a full tablet report...
I20260927 02:48:00.430652 18446 raft_consensus.cc:348] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.430918 18446 raft_consensus.cc:374] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:00.430984 18446 raft_consensus.cc:729] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2086076ff049b28e5ebf90c0d20fa6, State: Initialized, Role: FOLLOWER
W20260927 02:48:00.431685 18297 tablet.cc:2369] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:00.431486 18446 consensus_queue.cc:260] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.432588 18446 ts_tablet_manager.cc:1428] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:48:00.432935 18165 heartbeater.cc:503] Master 127.0.98.62:36903 was elected leader, sending a full tablet report...
W20260927 02:48:00.481750 18166 tablet.cc:2369] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:00.594059 18452 raft_consensus.cc:482] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:00.594314 18452 raft_consensus.cc:504] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.595542 18452 leader_election.cc:288] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d12cb2439fcc4cd8b54d9fac8c474183 (127.0.98.2:35043), 55b37c388f4143c18b024a1b925b2393 (127.0.98.3:41101)
I20260927 02:48:00.599576 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.596038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34145 (local address 127.0.98.3:41101)
0927 02:48:00.596152 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.596157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.596413 (+ 256us) server_negotiation.cc:407] Connection header received
0927 02:48:00.596514 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.596516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.596586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.596665 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:00.596968 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.597449 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.598561 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.598966 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.599044 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.599132 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.599372 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.599447 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:48:00.599596 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.595782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35043 (local address 127.0.98.1:38233)
0927 02:48:00.596150 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.596167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:00.596295 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:00.596582 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:00.596585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:00.596601 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:00.596751 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.596758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.597533 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.597539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.598459 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.598468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.599160 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.599163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.599274 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.599291 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:48:00.599385 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:48:00.599448 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":49,"thread_start_us":123,"threads_started":1}
I20260927 02:48:00.599601 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.595980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38233 (local address 127.0.98.2:35043)
0927 02:48:00.596119 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.596125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.596264 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:48:00.596408 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.596411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.596473 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.596556 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:00.596878 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.597415 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.598607 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.598988 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.599115 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.599171 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.599422 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.599499 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:48:00.599614 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.595813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41101 (local address 127.0.98.1:34145)
0927 02:48:00.596269 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.596284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:00.596415 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:00.596709 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:00.596711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:00.596725 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:00.596859 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.596866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.597591 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.597593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.598440 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:00.598448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.599093 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.599096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:00.599214 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.599270 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 02:48:00.599330 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:48:00.599419 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":32,"thread_start_us":146,"threads_started":1}
I20260927 02:48:00.600221 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173adb7dae8a4f7ebde713ca2e72717c" candidate_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" is_pre_election: true
I20260927 02:48:00.600458 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173adb7dae8a4f7ebde713ca2e72717c" candidate_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55b37c388f4143c18b024a1b925b2393" is_pre_election: true
I20260927 02:48:00.600538 18250 raft_consensus.cc:2454] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2086076ff049b28e5ebf90c0d20fa6 in term 0.
I20260927 02:48:00.600723 18381 raft_consensus.cc:2454] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2086076ff049b28e5ebf90c0d20fa6 in term 0.
I20260927 02:48:00.601051 18055 leader_election.cc:302] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d12cb2439fcc4cd8b54d9fac8c474183, fc2086076ff049b28e5ebf90c0d20fa6; no voters:
I20260927 02:48:00.601357 18452 raft_consensus.cc:2781] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:00.601444 18452 raft_consensus.cc:482] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:00.601477 18452 raft_consensus.cc:3037] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:00.602504 18452 raft_consensus.cc:504] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.602867 18452 leader_election.cc:288] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [CANDIDATE]: Term 1 election: Requested vote from peers d12cb2439fcc4cd8b54d9fac8c474183 (127.0.98.2:35043), 55b37c388f4143c18b024a1b925b2393 (127.0.98.3:41101)
I20260927 02:48:00.603235 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173adb7dae8a4f7ebde713ca2e72717c" candidate_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55b37c388f4143c18b024a1b925b2393"
I20260927 02:48:00.603253 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173adb7dae8a4f7ebde713ca2e72717c" candidate_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12cb2439fcc4cd8b54d9fac8c474183"
I20260927 02:48:00.603375 18381 raft_consensus.cc:3037] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:00.603384 18250 raft_consensus.cc:3037] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:00.604482 18381 raft_consensus.cc:2454] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc2086076ff049b28e5ebf90c0d20fa6 in term 1.
I20260927 02:48:00.604496 18250 raft_consensus.cc:2454] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc2086076ff049b28e5ebf90c0d20fa6 in term 1.
I20260927 02:48:00.604775 18055 leader_election.cc:302] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d12cb2439fcc4cd8b54d9fac8c474183, fc2086076ff049b28e5ebf90c0d20fa6; no voters:
I20260927 02:48:00.604962 18452 raft_consensus.cc:2781] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:00.605177 18452 raft_consensus.cc:686] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [term 1 LEADER]: Becoming Leader. State: Replica: fc2086076ff049b28e5ebf90c0d20fa6, State: Running, Role: LEADER
I20260927 02:48:00.605492 18452 consensus_queue.cc:237] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } }
I20260927 02:48:00.608626 17977 catalog_manager.cc:5495] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc2086076ff049b28e5ebf90c0d20fa6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2086076ff049b28e5ebf90c0d20fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 } health_report { overall_health: UNKNOWN } } }
W20260927 02:48:00.613659 18428 tablet.cc:2369] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:00.643103 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.639205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.0.98.1:41987)
0927 02:48:00.639360 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:00.639376 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:00.639393 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:48:00.639588 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:00.639590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:00.639647 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:00.639789 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:00.640091 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.640677 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.641685 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:00.641905 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:00.642015 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:00.642313 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:00.642711 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:00.642848 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:00.642909 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:48:00.642958 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 02:48:00.649025 18250 raft_consensus.cc:1261] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 1 FOLLOWER]: Refusing update from remote peer fc2086076ff049b28e5ebf90c0d20fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:00.649010 18381 raft_consensus.cc:1261] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 1 FOLLOWER]: Refusing update from remote peer fc2086076ff049b28e5ebf90c0d20fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:00.649753 18455 consensus_queue.cc:1035] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55b37c388f4143c18b024a1b925b2393" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:00.649991 18452 consensus_queue.cc:1035] T 173adb7dae8a4f7ebde713ca2e72717c P fc2086076ff049b28e5ebf90c0d20fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12cb2439fcc4cd8b54d9fac8c474183" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:00.660006 18459 mvcc.cc:205] Tried to move back new op lower bound from 7333794941531586560 to 7333794941362421760. Current Snapshot: MvccSnapshot[applied={T|T < 7333794941531586560 or (T in {7333794941531586560})}]
I20260927 02:48:00.667062 17976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:00.667237 17976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:00.669930 17976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9e75bb2d1eb4cbf871020041ccd0089: Sending DeleteTablet for 3 replicas of tablet 173adb7dae8a4f7ebde713ca2e72717c
I20260927 02:48:00.670699 18098 tablet_service.cc:1514] Processing DeleteTablet for tablet 173adb7dae8a4f7ebde713ca2e72717c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:00 UTC) from {username='slave'} at 127.0.0.1:46124
I20260927 02:48:00.670804 18361 tablet_service.cc:1514] Processing DeleteTablet for tablet 173adb7dae8a4f7ebde713ca2e72717c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:00 UTC) from {username='slave'} at 127.0.0.1:47230
I20260927 02:48:00.671675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 18037
I20260927 02:48:00.672689 18230 tablet_service.cc:1514] Processing DeleteTablet for tablet 173adb7dae8a4f7ebde713ca2e72717c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:00 UTC) from {username='slave'} at 127.0.0.1:57724
I20260927 02:48:00.672864 18469 tablet_replica.cc:330] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: stopping tablet replica
I20260927 02:48:00.673199 18469 raft_consensus.cc:2229] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:00.673511 18469 raft_consensus.cc:2258] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:00.674127 18470 tablet_replica.cc:330] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: stopping tablet replica
I20260927 02:48:00.674492 18470 raft_consensus.cc:2229] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:00.674840 18470 raft_consensus.cc:2258] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:00.675439 18469 ts_tablet_manager.cc:1905] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:00.676088 18470 ts_tablet_manager.cc:1905] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:00.678138 18470 ts_tablet_manager.cc:1918] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:00.678241 18470 log.cc:1196] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/173adb7dae8a4f7ebde713ca2e72717c
I20260927 02:48:00.678579 18470 ts_tablet_manager.cc:1939] T 173adb7dae8a4f7ebde713ca2e72717c P d12cb2439fcc4cd8b54d9fac8c474183: Deleting consensus metadata
I20260927 02:48:00.678975 18469 ts_tablet_manager.cc:1918] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:00.679072 18469 log.cc:1196] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/173adb7dae8a4f7ebde713ca2e72717c
I20260927 02:48:00.679404 18469 ts_tablet_manager.cc:1939] T 173adb7dae8a4f7ebde713ca2e72717c P 55b37c388f4143c18b024a1b925b2393: Deleting consensus metadata
I20260927 02:48:00.679735 17964 catalog_manager.cc:4841] TS d12cb2439fcc4cd8b54d9fac8c474183 (127.0.98.2:35043): tablet 173adb7dae8a4f7ebde713ca2e72717c (table test-table [id=aa702693d86e45ae955e7f421985d294]) successfully deleted
I20260927 02:48:00.680635 17961 catalog_manager.cc:4841] TS 55b37c388f4143c18b024a1b925b2393 (127.0.98.3:41101): tablet 173adb7dae8a4f7ebde713ca2e72717c (table test-table [id=aa702693d86e45ae955e7f421985d294]) successfully deleted
W20260927 02:48:00.680661 17962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41987 (error 108)
I20260927 02:48:00.680869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 18168
I20260927 02:48:00.682219 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.681966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41987 (local address 127.0.0.1:46134)
0927 02:48:00.682087 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.682154 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:48:00.682396 17962 catalog_manager.cc:4584] TS fc2086076ff049b28e5ebf90c0d20fa6 (127.0.98.1:41987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 173adb7dae8a4f7ebde713ca2e72717c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41987: connect: Connection refused (error 111)
I20260927 02:48:00.686398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 18299
I20260927 02:48:00.691740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 17946
2026-09-27T02:48:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1174 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:00 dist-test-slave-gkrl krb5kdc[18474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:00 dist-test-slave-gkrl krb5kdc[18474](info): set up 2 sockets
Sep 27 02:48:00 dist-test-slave-gkrl krb5kdc[18474](info): commencing operation
krb5kdc: starting...
W20260927 02:48:01.727751 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:01 dist-test-slave-gkrl krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:01.847162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:01.957577 18487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:01.957844 18487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:01.957882 18487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:01.961758 18487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:01.961807 18487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:01.961826 18487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:01.961843 18487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:01.961858 18487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:01.966862 18487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:01.967774 18487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:01.968722 18487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:01.977010 18492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:01.977012 18495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:01.976985 18493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:01.977065 18487 server_base.cc:1034] running on GCE node
I20260927 02:48:01.977826 18487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:01.978858 18487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:01.980037 18487 hybrid_clock.cc:648] HybridClock initialized: now 1790477281980026 us; error 26 us; skew 500 ppm
Sep 27 02:48:01 dist-test-slave-gkrl krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:01.982910 18487 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:01.983819 18487 webserver.cc:460] Webserver started at http://127.0.98.62:44379/ using document root <none> and password file <none>
I20260927 02:48:01.984448 18487 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:01.984556 18487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:01.984795 18487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:01.986835 18487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "35a9a6d7cadc4deca8a03cf84df85282"
format_stamp: "Formatted at 2026-09-27 02:48:01 on dist-test-slave-gkrl"
server_key: "17cf58d112d6f3c2b126d16c863f2e97"
server_key_iv: "eab4d1b3b939aeb75193d96993b61861"
server_key_version: "encryptionkey@0"
I20260927 02:48:01.987416 18487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "35a9a6d7cadc4deca8a03cf84df85282"
format_stamp: "Formatted at 2026-09-27 02:48:01 on dist-test-slave-gkrl"
server_key: "17cf58d112d6f3c2b126d16c863f2e97"
server_key_iv: "eab4d1b3b939aeb75193d96993b61861"
server_key_version: "encryptionkey@0"
I20260927 02:48:01.991104 18487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:48:01.993685 18502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:01.994977 18487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:01.995123 18487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "35a9a6d7cadc4deca8a03cf84df85282"
format_stamp: "Formatted at 2026-09-27 02:48:01 on dist-test-slave-gkrl"
server_key: "17cf58d112d6f3c2b126d16c863f2e97"
server_key_iv: "eab4d1b3b939aeb75193d96993b61861"
server_key_version: "encryptionkey@0"
I20260927 02:48:01.995234 18487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:02.009968 18487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:02.013072 18487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:02.013267 18487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:02.020341 18487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37011
I20260927 02:48:02.020362 18554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37011 every 8 connection(s)
I20260927 02:48:02.021332 18487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:02.023208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 18487
I20260927 02:48:02.023355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:02.023674 392 external_mini_cluster.cc:1428] Setting key 3de572fb38fcd9e89b0cfb46ac1504bd
I20260927 02:48:02.024405 18555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:02.030985 18555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282: Bootstrap starting.
I20260927 02:48:02.033211 18555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:02.033912 18555 log.cc:824] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:02.035934 18555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282: No bootstrap required, opened a new log
Sep 27 02:48:02 dist-test-slave-gkrl krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:02.039090 18555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a9a6d7cadc4deca8a03cf84df85282" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260927 02:48:02.039301 18555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:02.039336 18555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a9a6d7cadc4deca8a03cf84df85282, State: Initialized, Role: FOLLOWER
I20260927 02:48:02.039806 18555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a9a6d7cadc4deca8a03cf84df85282" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260927 02:48:02.040014 18555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:02.040099 18555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:02.040210 18555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:02.041335 18555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a9a6d7cadc4deca8a03cf84df85282" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260927 02:48:02.041667 18555 leader_election.cc:302] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35a9a6d7cadc4deca8a03cf84df85282; no voters:
I20260927 02:48:02.042011 18555 leader_election.cc:288] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:02.042109 18560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:02.042313 18560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [term 1 LEADER]: Becoming Leader. State: Replica: 35a9a6d7cadc4deca8a03cf84df85282, State: Running, Role: LEADER
I20260927 02:48:02.042623 18560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a9a6d7cadc4deca8a03cf84df85282" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260927 02:48:02.043284 18555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:02.043697 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35a9a6d7cadc4deca8a03cf84df85282. Latest consensus state: current_term: 1 leader_uuid: "35a9a6d7cadc4deca8a03cf84df85282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a9a6d7cadc4deca8a03cf84df85282" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260927 02:48:02.043821 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:02.043949 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.025056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.0.98.62:37011)
0927 02:48:02.025462 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.025486 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.025522 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:48:02.026138 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.026153 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.026485 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.026758 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:02.027713 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.028762 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.029993 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.030452 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.030564 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.037233 (+ 6669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:02.037238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:02.037251 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:02.037279 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:02.040295 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.040809 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.040814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.040818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.040942 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.041253 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.041255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.041256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.041718 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:02.042013 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.042643 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.042934 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":137,"threads_started":1}
I20260927 02:48:02.044243 18564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:02.044556 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35a9a6d7cadc4deca8a03cf84df85282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a9a6d7cadc4deca8a03cf84df85282" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260927 02:48:02.044700 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:02.047340 18564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:02.052738 18564 catalog_manager.cc:1347] Generated new cluster ID: 86ad7b07d3d24243b7286e83654d3d7d
I20260927 02:48:02.052811 18564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:02.064776 18564 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:02.066030 18564 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:02.078660 18564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282: Generated new TSK 0
I20260927 02:48:02.079535 18564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:48:02.119243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:02.230151 18581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:02.230459 18581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:02.230537 18581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:02.234556 18581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:02.234649 18581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:02.234745 18581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:02.239929 18581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:02.240983 18581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:02.242007 18581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:02.252247 18589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:02.252274 18587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:02.252477 18581 server_base.cc:1034] running on GCE node
W20260927 02:48:02.252192 18586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:02.252956 18581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:02.253664 18581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:02.255013 18581 hybrid_clock.cc:648] HybridClock initialized: now 1790477282254818 us; error 48 us; skew 500 ppm
Sep 27 02:48:02 dist-test-slave-gkrl krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:02.257934 18581 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:48:02.258857 18581 webserver.cc:460] Webserver started at http://127.0.98.1:43501/ using document root <none> and password file <none>
I20260927 02:48:02.259379 18581 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:02.259443 18581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:02.259609 18581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:02.261873 18581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "c653868213f892b385d7da19558f8af5"
server_key_iv: "a521919d1f83d28786303f715c768d47"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.262382 18581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "c653868213f892b385d7da19558f8af5"
server_key_iv: "a521919d1f83d28786303f715c768d47"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.266314 18581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 02:48:02.268698 18596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.269948 18581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:02.270038 18581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "c653868213f892b385d7da19558f8af5"
server_key_iv: "a521919d1f83d28786303f715c768d47"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.270120 18581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:02.285943 18581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:02.288890 18581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:02.289037 18581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:02.289539 18581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:02.290311 18581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:02.290357 18581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.290395 18581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:02.290419 18581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.299551 18581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42449
I20260927 02:48:02.299578 18709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42449 every 8 connection(s)
I20260927 02:48:02.300544 18581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:48:02.306061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 18581
I20260927 02:48:02.306232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:02.306625 392 external_mini_cluster.cc:1428] Setting key ec79aca839d2b899affdf0337fa5a0df
Sep 27 02:48:02 dist-test-slave-gkrl krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:02.315109 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.302440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52569 (local address 127.0.98.62:37011)
0927 02:48:02.302597 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.302601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.303589 (+ 988us) server_negotiation.cc:407] Connection header received
0927 02:48:02.304464 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.304467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.304514 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.304591 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:02.305722 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.306330 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.307870 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.308232 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.308316 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.311431 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:02.311435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:02.311437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:02.311459 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:02.313012 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.313586 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.313591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.313593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.313672 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.314034 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.314037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.314039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.314217 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:02.314323 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.314799 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.314937 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 02:48:02.315958 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.302785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.1:52569)
0927 02:48:02.303445 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.303484 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:02.304268 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:02.304760 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:02.304767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:02.305104 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:02.305592 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.305604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.306481 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.306486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.307738 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.307750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.308406 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.308411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.308611 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.309300 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:02.309327 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:02.311278 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:02.313135 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:02.313141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:02.313156 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:02.313479 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:02.313765 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:02.313769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:02.313771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:02.313928 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:02.314320 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:02.314326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:02.314571 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 02:48:02.314762 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 02:48:02.315080 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:48:02.317257 18710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260927 02:48:02.317536 18710 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:02.318078 18710 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:48:02.319679 18519 ts_manager.cc:194] Registered new tserver with Master: 6dd98ce0c88e4d7d9bee4d36726dcf7b (127.0.98.1:42449)
I20260927 02:48:02.321070 18519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:48:02.335562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:02.442770 18715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:02.443028 18715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:02.443070 18715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:02.447239 18715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:02.447297 18715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:02.447358 18715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:02.452479 18715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:02.453424 18715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:02.454346 18715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:02.463824 18721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:02.463814 18720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:02.463897 18723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:02.463961 18715 server_base.cc:1034] running on GCE node
I20260927 02:48:02.464524 18715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:02.465159 18715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:02.466337 18715 hybrid_clock.cc:648] HybridClock initialized: now 1790477282466324 us; error 25 us; skew 500 ppm
Sep 27 02:48:02 dist-test-slave-gkrl krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:02.469139 18715 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:48:02.469976 18715 webserver.cc:460] Webserver started at http://127.0.98.2:43749/ using document root <none> and password file <none>
I20260927 02:48:02.470508 18715 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:02.470597 18715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:02.470829 18715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:02.472851 18715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2ae21325717461b92cfc0d610bfc6a9"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "00a86adc85f7724365040580a5559ef1"
server_key_iv: "a1dabacc8ff21551cb0976356e6f771d"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.473390 18715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2ae21325717461b92cfc0d610bfc6a9"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "00a86adc85f7724365040580a5559ef1"
server_key_iv: "a1dabacc8ff21551cb0976356e6f771d"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.476976 18715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:48:02.479379 18730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.480638 18715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:48:02.480762 18715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "d2ae21325717461b92cfc0d610bfc6a9"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "00a86adc85f7724365040580a5559ef1"
server_key_iv: "a1dabacc8ff21551cb0976356e6f771d"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.480865 18715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:02.497167 18715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:02.500597 18715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:02.500790 18715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:02.501332 18715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:02.502121 18715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:02.502187 18715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.502254 18715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:02.502300 18715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.511504 18715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44593
I20260927 02:48:02.511567 18843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44593 every 8 connection(s)
I20260927 02:48:02.512486 18715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:48:02 dist-test-slave-gkrl krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:02.521675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 18715
I20260927 02:48:02.521816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:02.522184 392 external_mini_cluster.cc:1428] Setting key 2a8240f6afdd58694f2e2faa8f7fb4db
I20260927 02:48:02.525890 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.514387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36849 (local address 127.0.98.62:37011)
0927 02:48:02.514522 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.514527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.515359 (+ 832us) server_negotiation.cc:407] Connection header received
0927 02:48:02.516309 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.516312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.516367 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.516450 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:02.517589 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.518132 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.519271 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.519500 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.519615 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.522123 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:02.522128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:02.522131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:02.522161 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:02.523710 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.524220 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.524225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.524230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.524331 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.524671 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.524677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.524682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.524949 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:02.525049 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.525587 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.525714 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:48:02.526839 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.514752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.2:36849)
0927 02:48:02.515198 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.515238 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:02.516107 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:02.516618 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:02.516624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:02.516977 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:02.517441 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.517456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.518273 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.518281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.519147 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.519156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.519650 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.519653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.519820 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.520374 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:02.520390 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:02.521936 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:02.523822 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:02.523826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:02.523839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:02.524118 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:02.524426 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:02.524428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:02.524429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:02.524570 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:02.525065 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:02.525071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:02.525317 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 02:48:02.525519 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 02:48:02.525865 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260927 02:48:02.528075 18844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260927 02:48:02.528352 18844 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:02.528988 18844 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260927 02:48:02.530177 18519 ts_manager.cc:194] Registered new tserver with Master: d2ae21325717461b92cfc0d610bfc6a9 (127.0.98.2:44593)
I20260927 02:48:02.531001 18519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36849
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:48:02.551069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:02.659933 18849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:02.660193 18849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:02.660236 18849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:02.664294 18849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:02.664353 18849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:02.664435 18849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:02.669674 18849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:02.670711 18849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:02.671691 18849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:02.681386 18854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:02.681469 18849 server_base.cc:1034] running on GCE node
W20260927 02:48:02.681408 18857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:02.681396 18855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:02.682006 18849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:02.682587 18849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:02.683753 18849 hybrid_clock.cc:648] HybridClock initialized: now 1790477282683723 us; error 41 us; skew 500 ppm
Sep 27 02:48:02 dist-test-slave-gkrl krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:02.686427 18849 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:48:02.687266 18849 webserver.cc:460] Webserver started at http://127.0.98.3:33001/ using document root <none> and password file <none>
I20260927 02:48:02.687762 18849 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:02.687817 18849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:02.688050 18849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:02.689968 18849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "010f45f970de44bba96b8285b96d77b6"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "623e62dc519e6f2b2d2d03e320937e00"
server_key_iv: "77b18cc4fc1b47aee24dedee1b9fcb5a"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.690451 18849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "010f45f970de44bba96b8285b96d77b6"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "623e62dc519e6f2b2d2d03e320937e00"
server_key_iv: "77b18cc4fc1b47aee24dedee1b9fcb5a"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.694011 18849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:48:02.696400 18864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.697757 18849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:02.697922 18849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "010f45f970de44bba96b8285b96d77b6"
format_stamp: "Formatted at 2026-09-27 02:48:02 on dist-test-slave-gkrl"
server_key: "623e62dc519e6f2b2d2d03e320937e00"
server_key_iv: "77b18cc4fc1b47aee24dedee1b9fcb5a"
server_key_version: "encryptionkey@0"
I20260927 02:48:02.698035 18849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:02.722373 18849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:02.725241 18849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:02.725445 18849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:02.726024 18849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:02.726894 18849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:02.726961 18849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.727029 18849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:02.727082 18849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:02.736193 18849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41897
I20260927 02:48:02.736236 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41897 every 8 connection(s)
I20260927 02:48:02.737128 18849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:48:02 dist-test-slave-gkrl krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:02.747231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 18849
I20260927 02:48:02.747355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:02.747738 392 external_mini_cluster.cc:1428] Setting key 481448f67bb44501070729c90ab9542a
I20260927 02:48:02.750559 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.738972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43605 (local address 127.0.98.62:37011)
0927 02:48:02.739099 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.739110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.739915 (+ 805us) server_negotiation.cc:407] Connection header received
0927 02:48:02.740808 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.740811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.740870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.740978 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:02.742104 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.742640 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.743848 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.744102 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.744217 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.746645 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:02.746648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:02.746650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:02.746675 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:02.748329 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.748921 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.748924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.748925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.749055 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.749449 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.749453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.749454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.749676 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:02.749781 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.750345 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.750447 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:48:02.751379 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.739296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.3:43605)
0927 02:48:02.739735 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.739775 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:02.740603 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:02.741146 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:02.741153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:02.741496 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:02.741957 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.741970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.742768 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.742774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.743723 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.743733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.744252 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.744256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.744444 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.745041 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:02.745056 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:02.746498 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:02.748436 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:02.748441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:02.748462 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:02.748800 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:02.749155 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:02.749158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:02.749160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:02.749343 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:02.749776 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:02.749781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:02.750104 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 02:48:02.750284 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 02:48:02.750538 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260927 02:48:02.752440 18978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260927 02:48:02.752708 18978 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:02.753170 18978 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260927 02:48:02.754145 18519 ts_manager.cc:194] Registered new tserver with Master: 010f45f970de44bba96b8285b96d77b6 (127.0.98.3:41897)
I20260927 02:48:02.754693 18519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43605
I20260927 02:48:02.761504 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:48:02.770704 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.763430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.0.98.62:37011)
0927 02:48:02.763561 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.763565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.763725 (+ 160us) server_negotiation.cc:407] Connection header received
0927 02:48:02.763803 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.763805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.763839 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.763977 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:02.764583 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.765067 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.765855 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.766072 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.766148 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.767537 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:02.767540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:02.767542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:02.767563 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:02.769075 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.769552 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.769555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.769556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.769624 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:02.770010 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:02.770012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:02.770013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:02.770187 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:02.770295 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.770526 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.770596 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:48:02.773518 18519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:48:02.776604 18519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:02.790536 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.784785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.0.98.1:42449)
0927 02:48:02.785575 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.785591 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.786192 (+ 601us) server_negotiation.cc:407] Connection header received
0927 02:48:02.786370 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.786375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.786580 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.786733 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:02.787164 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.787973 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.789311 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.789866 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.789988 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.790138 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.790301 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.790395 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":81,"threads_started":1}
I20260927 02:48:02.790532 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.784668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42449 (local address 127.0.0.1:55380)
0927 02:48:02.786063 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.786099 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:02.786207 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:02.786932 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:02.786936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:02.786955 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:02.787063 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.787070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.788123 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.788128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.789185 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.789195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.789987 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.789990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.790091 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.790126 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:48:02.790258 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 02:48:02.790379 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"spinlock_wait_cycles":5376,"thread_start_us":99,"threads_started":1}
I20260927 02:48:02.791481 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.785026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44593 (local address 127.0.0.1:44936)
0927 02:48:02.786423 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.786435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:02.786519 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:02.786954 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:02.786957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:02.786976 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:02.787078 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.787086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.788453 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.788456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.789296 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.789306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.791094 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.791101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.791194 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.791217 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:48:02.791293 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:48:02.791358 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260927 02:48:02.791694 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.784782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41897 (local address 127.0.0.1:42414)
0927 02:48:02.786067 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.786100 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:02.786216 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:02.786741 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:02.786747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:02.786767 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:02.786911 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.786920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.788425 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.788431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.789994 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.790001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.791291 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.791294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.791388 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.791422 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:48:02.791485 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:48:02.791551 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"mutex_wait_us":829,"thread_start_us":81,"threads_started":1}
I20260927 02:48:02.792181 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.785504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.0.98.3:41897)
0927 02:48:02.785972 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.785984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.786251 (+ 267us) server_negotiation.cc:407] Connection header received
0927 02:48:02.786338 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.786345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.786577 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.786751 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:02.787038 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.788254 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.790112 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.790939 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.791088 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.791844 (+ 756us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.791945 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.792045 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260927 02:48:02.792263 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.785488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.0.98.2:44593)
0927 02:48:02.785928 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.785941 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.786524 (+ 583us) server_negotiation.cc:407] Connection header received
0927 02:48:02.786606 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.786611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.786780 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.786972 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:02.787212 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.788320 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.790178 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.790943 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.791088 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.791222 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.792021 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.792117 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":82,"threads_started":1}
I20260927 02:48:02.794123 18644 tablet_service.cc:1467] Processing CreateTablet for tablet 97c402fd1b95461ea00a15ae7759af3e (DEFAULT_TABLE table=test-table [id=12dd851f4df54174ab1c01e6d2085ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:02.794399 18912 tablet_service.cc:1467] Processing CreateTablet for tablet 97c402fd1b95461ea00a15ae7759af3e (DEFAULT_TABLE table=test-table [id=12dd851f4df54174ab1c01e6d2085ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:02.794745 18778 tablet_service.cc:1467] Processing CreateTablet for tablet 97c402fd1b95461ea00a15ae7759af3e (DEFAULT_TABLE table=test-table [id=12dd851f4df54174ab1c01e6d2085ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:02.795342 18644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c402fd1b95461ea00a15ae7759af3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:02.795477 18912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c402fd1b95461ea00a15ae7759af3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:02.795624 18778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c402fd1b95461ea00a15ae7759af3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:02.802443 18997 tablet_bootstrap.cc:492] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: Bootstrap starting.
I20260927 02:48:02.803104 18995 tablet_bootstrap.cc:492] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: Bootstrap starting.
I20260927 02:48:02.804466 18996 tablet_bootstrap.cc:492] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b: Bootstrap starting.
I20260927 02:48:02.805208 18995 tablet_bootstrap.cc:654] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:02.805265 18997 tablet_bootstrap.cc:654] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:02.806134 18997 log.cc:824] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:02.806182 18995 log.cc:824] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:02.806684 18996 tablet_bootstrap.cc:654] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:02.807899 18996 log.cc:824] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:02.808291 18997 tablet_bootstrap.cc:492] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: No bootstrap required, opened a new log
I20260927 02:48:02.808321 18995 tablet_bootstrap.cc:492] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: No bootstrap required, opened a new log
I20260927 02:48:02.808506 18997 ts_tablet_manager.cc:1397] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:48:02.808508 18995 ts_tablet_manager.cc:1397] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:48:02.809829 18996 tablet_bootstrap.cc:492] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b: No bootstrap required, opened a new log
I20260927 02:48:02.810060 18996 ts_tablet_manager.cc:1397] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:48:02.811571 18995 raft_consensus.cc:348] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.811825 18995 raft_consensus.cc:374] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:02.811878 18995 raft_consensus.cc:729] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010f45f970de44bba96b8285b96d77b6, State: Initialized, Role: FOLLOWER
I20260927 02:48:02.812333 18995 consensus_queue.cc:260] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.812306 18997 raft_consensus.cc:348] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.812567 18997 raft_consensus.cc:374] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:02.812628 18997 raft_consensus.cc:729] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ae21325717461b92cfc0d610bfc6a9, State: Initialized, Role: FOLLOWER
I20260927 02:48:02.813081 18997 consensus_queue.cc:260] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.813282 18995 ts_tablet_manager.cc:1428] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:48:02.813620 18978 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
I20260927 02:48:02.813639 18844 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
I20260927 02:48:02.814009 18997 ts_tablet_manager.cc:1428] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:48:02.813908 18996 raft_consensus.cc:348] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.814136 18996 raft_consensus.cc:374] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:02.814184 18996 raft_consensus.cc:729] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd98ce0c88e4d7d9bee4d36726dcf7b, State: Initialized, Role: FOLLOWER
I20260927 02:48:02.814576 18996 consensus_queue.cc:260] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.815227 18710 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
I20260927 02:48:02.816345 18996 ts_tablet_manager.cc:1428] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:48:02.861142 19003 raft_consensus.cc:482] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:02.861375 19003 raft_consensus.cc:504] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.862627 19003 leader_election.cc:288] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 010f45f970de44bba96b8285b96d77b6 (127.0.98.3:41897), d2ae21325717461b92cfc0d610bfc6a9 (127.0.98.2:44593)
I20260927 02:48:02.866888 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.862950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34481 (local address 127.0.98.3:41897)
0927 02:48:02.863086 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.863093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.863276 (+ 183us) server_negotiation.cc:407] Connection header received
0927 02:48:02.863457 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.863461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.863507 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.863587 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:02.863861 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.864394 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.865896 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.866297 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.866429 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.866492 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.866676 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.866748 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:48:02.866926 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.863091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39905 (local address 127.0.98.2:44593)
0927 02:48:02.863214 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.863219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.863468 (+ 249us) server_negotiation.cc:407] Connection header received
0927 02:48:02.863606 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.863609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.863665 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.863759 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:02.864070 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.864603 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.865940 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.866347 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.866486 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.866552 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.866734 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.866802 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:48:02.866952 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.862803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41897 (local address 127.0.98.1:34481)
0927 02:48:02.863196 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.863211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:02.863321 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:02.863610 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:02.863613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:02.863629 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:02.863751 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.863756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.864531 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.864534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.865766 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.865775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.866435 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.866439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.866537 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.866560 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:48:02.866659 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:48:02.866733 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":56,"thread_start_us":133,"threads_started":1}
I20260927 02:48:02.867307 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.862997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44593 (local address 127.0.98.1:39905)
0927 02:48:02.863360 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.863371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:02.863493 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:02.863769 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:02.863772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:02.863788 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:02.863958 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.863965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.864729 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.864732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.865827 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:02.865836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.866486 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.866490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:02.866597 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.866618 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:48:02.866712 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:48:02.866784 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260927 02:48:02.867570 18932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c402fd1b95461ea00a15ae7759af3e" candidate_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010f45f970de44bba96b8285b96d77b6" is_pre_election: true
I20260927 02:48:02.867964 18932 raft_consensus.cc:2454] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd98ce0c88e4d7d9bee4d36726dcf7b in term 0.
I20260927 02:48:02.867988 18798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c402fd1b95461ea00a15ae7759af3e" candidate_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ae21325717461b92cfc0d610bfc6a9" is_pre_election: true
I20260927 02:48:02.868273 18798 raft_consensus.cc:2454] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd98ce0c88e4d7d9bee4d36726dcf7b in term 0.
I20260927 02:48:02.868507 18597 leader_election.cc:302] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010f45f970de44bba96b8285b96d77b6, 6dd98ce0c88e4d7d9bee4d36726dcf7b; no voters:
I20260927 02:48:02.868835 19003 raft_consensus.cc:2781] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:02.868950 19003 raft_consensus.cc:482] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:02.869012 19003 raft_consensus.cc:3037] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:02.870050 19003 raft_consensus.cc:504] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.870443 19003 leader_election.cc:288] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [CANDIDATE]: Term 1 election: Requested vote from peers 010f45f970de44bba96b8285b96d77b6 (127.0.98.3:41897), d2ae21325717461b92cfc0d610bfc6a9 (127.0.98.2:44593)
I20260927 02:48:02.870800 18798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c402fd1b95461ea00a15ae7759af3e" candidate_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ae21325717461b92cfc0d610bfc6a9"
I20260927 02:48:02.870908 18932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c402fd1b95461ea00a15ae7759af3e" candidate_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010f45f970de44bba96b8285b96d77b6"
I20260927 02:48:02.870959 18798 raft_consensus.cc:3037] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:02.871039 18932 raft_consensus.cc:3037] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:02.872308 18798 raft_consensus.cc:2454] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd98ce0c88e4d7d9bee4d36726dcf7b in term 1.
I20260927 02:48:02.872314 18932 raft_consensus.cc:2454] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd98ce0c88e4d7d9bee4d36726dcf7b in term 1.
I20260927 02:48:02.872625 18600 leader_election.cc:302] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd98ce0c88e4d7d9bee4d36726dcf7b, d2ae21325717461b92cfc0d610bfc6a9; no voters:
I20260927 02:48:02.872814 19003 raft_consensus.cc:2781] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:02.873037 19003 raft_consensus.cc:686] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 1 LEADER]: Becoming Leader. State: Replica: 6dd98ce0c88e4d7d9bee4d36726dcf7b, State: Running, Role: LEADER
I20260927 02:48:02.873404 19003 consensus_queue.cc:237] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } }
I20260927 02:48:02.876586 18515 catalog_manager.cc:5495] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd98ce0c88e4d7d9bee4d36726dcf7b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd98ce0c88e4d7d9bee4d36726dcf7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 } health_report { overall_health: UNKNOWN } } }
I20260927 02:48:02.893345 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.889756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.0.98.1:42449)
0927 02:48:02.889889 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:02.889894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:02.889909 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:48:02.890031 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:02.890033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:02.890092 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:02.890215 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:02.890465 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.890975 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.891849 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:02.892115 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:02.892233 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:02.892528 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:02.892917 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:02.893080 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:02.893167 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 02:48:02.893205 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:48:02.899286 18798 raft_consensus.cc:1261] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 1 FOLLOWER]: Refusing update from remote peer 6dd98ce0c88e4d7d9bee4d36726dcf7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:02.899286 18932 raft_consensus.cc:1261] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 1 FOLLOWER]: Refusing update from remote peer 6dd98ce0c88e4d7d9bee4d36726dcf7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:02.900187 19006 consensus_queue.cc:1035] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ae21325717461b92cfc0d610bfc6a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:02.900478 19003 consensus_queue.cc:1035] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "010f45f970de44bba96b8285b96d77b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:02.909404 19011 mvcc.cc:205] Tried to move back new op lower bound from 7333794950747635712 to 7333794950652395520. Current Snapshot: MvccSnapshot[applied={T|T < 7333794950747635712}]
I20260927 02:48:02.917734 18515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:02.917886 18515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:02.920269 18515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35a9a6d7cadc4deca8a03cf84df85282: Sending DeleteTablet for 3 replicas of tablet 97c402fd1b95461ea00a15ae7759af3e
I20260927 02:48:02.921062 18644 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c402fd1b95461ea00a15ae7759af3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55380
I20260927 02:48:02.921384 18912 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c402fd1b95461ea00a15ae7759af3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42414
I20260927 02:48:02.921595 19019 tablet_replica.cc:330] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b: stopping tablet replica
I20260927 02:48:02.921898 19020 tablet_replica.cc:330] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: stopping tablet replica
I20260927 02:48:02.921913 19019 raft_consensus.cc:2229] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:02.922021 18778 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c402fd1b95461ea00a15ae7759af3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44936
I20260927 02:48:02.922322 19019 raft_consensus.cc:2258] T 97c402fd1b95461ea00a15ae7759af3e P 6dd98ce0c88e4d7d9bee4d36726dcf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:02.922340 19020 raft_consensus.cc:2229] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:02.922753 19020 raft_consensus.cc:2258] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:02.922765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 18581
I20260927 02:48:02.922794 19021 tablet_replica.cc:330] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: stopping tablet replica
I20260927 02:48:02.923564 19021 raft_consensus.cc:2229] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:02.924343 19021 raft_consensus.cc:2258] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:02.925434 19020 ts_tablet_manager.cc:1905] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:02.925696 19021 ts_tablet_manager.cc:1905] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:02.927759 19021 ts_tablet_manager.cc:1918] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:02.927877 19021 log.cc:1196] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/97c402fd1b95461ea00a15ae7759af3e
I20260927 02:48:02.928200 19021 ts_tablet_manager.cc:1939] T 97c402fd1b95461ea00a15ae7759af3e P d2ae21325717461b92cfc0d610bfc6a9: Deleting consensus metadata
I20260927 02:48:02.928181 19020 ts_tablet_manager.cc:1918] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:02.928288 19020 log.cc:1196] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/97c402fd1b95461ea00a15ae7759af3e
I20260927 02:48:02.928593 19020 ts_tablet_manager.cc:1939] T 97c402fd1b95461ea00a15ae7759af3e P 010f45f970de44bba96b8285b96d77b6: Deleting consensus metadata
I20260927 02:48:02.929366 18506 catalog_manager.cc:4841] TS d2ae21325717461b92cfc0d610bfc6a9 (127.0.98.2:44593): tablet 97c402fd1b95461ea00a15ae7759af3e (table test-table [id=12dd851f4df54174ab1c01e6d2085ed4]) successfully deleted
I20260927 02:48:02.929617 18503 catalog_manager.cc:4841] TS 010f45f970de44bba96b8285b96d77b6 (127.0.98.3:41897): tablet 97c402fd1b95461ea00a15ae7759af3e (table test-table [id=12dd851f4df54174ab1c01e6d2085ed4]) successfully deleted
W20260927 02:48:02.930163 18505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42449 (error 108)
I20260927 02:48:02.930440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 18715
I20260927 02:48:02.932443 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.932180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42449 (local address 127.0.0.1:55410)
0927 02:48:02.932316 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.932375 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 02:48:02.932677 18505 catalog_manager.cc:4584] TS 6dd98ce0c88e4d7d9bee4d36726dcf7b (127.0.98.1:42449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97c402fd1b95461ea00a15ae7759af3e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42449: connect: Connection refused (error 111)
I20260927 02:48:02.936110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 18849
I20260927 02:48:02.941777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 18487
2026-09-27T02:48:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2250 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T02:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:02Z Disabled control of system clock
I20260927 02:48:02.971544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41893
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:48:03.080806 19026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:03.081075 19026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:03.081110 19026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:03.085177 19026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:03.085256 19026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:03.085273 19026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:03.085290 19026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:03.085306 19026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:03.090359 19026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:03.091416 19026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:03.092778 19026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:03.101341 19032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:03.101284 19034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:03.101318 19031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:03.101606 19026 server_base.cc:1034] running on GCE node
I20260927 02:48:03.102118 19026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:03.103132 19026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:03.104314 19026 hybrid_clock.cc:648] HybridClock initialized: now 1790477283104284 us; error 45 us; skew 500 ppm
I20260927 02:48:03.106284 19026 webserver.cc:460] Webserver started at http://127.0.98.62:46317/ using document root <none> and password file <none>
I20260927 02:48:03.106920 19026 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:03.106983 19026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:03.107190 19026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:03.109197 19026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "7c7f943579c947f484c234e4bbda8c1b"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "0155bf57ab50ce932d539ccfa42f2a60"
server_key_iv: "d81d5fc46792ef5a9055700a687a6a1c"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.109730 19026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c7f943579c947f484c234e4bbda8c1b"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "0155bf57ab50ce932d539ccfa42f2a60"
server_key_iv: "d81d5fc46792ef5a9055700a687a6a1c"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.113245 19026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:48:03.115449 19040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.116649 19026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:03.116813 19026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "7c7f943579c947f484c234e4bbda8c1b"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "0155bf57ab50ce932d539ccfa42f2a60"
server_key_iv: "d81d5fc46792ef5a9055700a687a6a1c"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.116923 19026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:03.133972 19026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:03.134610 19026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:03.134806 19026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:03.141650 19026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41893
I20260927 02:48:03.141660 19092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41893 every 8 connection(s)
I20260927 02:48:03.142513 19026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:03.145514 19093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:03.146976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 19026
I20260927 02:48:03.147095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:03.147425 392 external_mini_cluster.cc:1428] Setting key 2b7f957d817ae4b90779b6e58e05004a
I20260927 02:48:03.151542 19093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b: Bootstrap starting.
I20260927 02:48:03.153657 19093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:03.154489 19093 log.cc:824] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:03.156260 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.148831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.0.98.62:41893)
0927 02:48:03.149427 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.149442 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.149475 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:48:03.150111 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.150126 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.150484 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.150810 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:03.151200 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.152253 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.153130 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.153581 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.153750 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.154570 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:03.154575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:03.154587 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:03.154982 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:03.155035 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:03.155054 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:03.155161 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.155307 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.155541 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260927 02:48:03.156416 19093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b: No bootstrap required, opened a new log
I20260927 02:48:03.159858 19093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f943579c947f484c234e4bbda8c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41893 } }
I20260927 02:48:03.160143 19093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:03.160221 19093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7f943579c947f484c234e4bbda8c1b, State: Initialized, Role: FOLLOWER
I20260927 02:48:03.160693 19093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f943579c947f484c234e4bbda8c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41893 } }
I20260927 02:48:03.160861 19093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:03.160933 19093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:03.161036 19093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:03.161860 19093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f943579c947f484c234e4bbda8c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41893 } }
I20260927 02:48:03.162137 19093 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7f943579c947f484c234e4bbda8c1b; no voters:
I20260927 02:48:03.162451 19093 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:03.162545 19098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:03.162777 19098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [term 1 LEADER]: Becoming Leader. State: Replica: 7c7f943579c947f484c234e4bbda8c1b, State: Running, Role: LEADER
I20260927 02:48:03.163058 19098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f943579c947f484c234e4bbda8c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41893 } }
I20260927 02:48:03.163712 19093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:03.164752 19099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7f943579c947f484c234e4bbda8c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f943579c947f484c234e4bbda8c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41893 } } }
I20260927 02:48:03.164883 19099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:03.165257 19107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:03.165210 19100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7f943579c947f484c234e4bbda8c1b. Latest consensus state: current_term: 1 leader_uuid: "7c7f943579c947f484c234e4bbda8c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f943579c947f484c234e4bbda8c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41893 } } }
I20260927 02:48:03.165731 19100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:03.167846 19107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:03.172991 19107 catalog_manager.cc:1347] Generated new cluster ID: f0fd5d7e52064b93929fbcb024989457
I20260927 02:48:03.173054 19107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:03.186498 19107 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:03.187373 19107 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:03.192602 19107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b: Generated new TSK 0
I20260927 02:48:03.193334 19107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:03.202200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41893
--builtin_ntp_servers=127.0.98.20:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:48:03.311717 19117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:03.312093 19117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:03.312175 19117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:03.316183 19117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:03.316249 19117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:03.316334 19117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:03.321084 19117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:03.322156 19117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:03.323493 19117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:03.333310 19125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:03.333304 19122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:03.333295 19123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:03.333353 19117 server_base.cc:1034] running on GCE node
I20260927 02:48:03.333927 19117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:03.334604 19117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:03.335793 19117 hybrid_clock.cc:648] HybridClock initialized: now 1790477283335758 us; error 47 us; skew 500 ppm
I20260927 02:48:03.337656 19117 webserver.cc:460] Webserver started at http://127.0.98.1:41639/ using document root <none> and password file <none>
I20260927 02:48:03.338238 19117 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:03.338347 19117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:03.338595 19117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:03.340692 19117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "01286026ca154525a4f825864eb44077"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "3d1bb490dc5497727353f476a49f17ca"
server_key_iv: "847f3d78695ba42fcb0b1a9b04fa1cb3"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.341235 19117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01286026ca154525a4f825864eb44077"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "3d1bb490dc5497727353f476a49f17ca"
server_key_iv: "847f3d78695ba42fcb0b1a9b04fa1cb3"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.344832 19117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:48:03.347182 19131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.348441 19117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:03.348574 19117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "01286026ca154525a4f825864eb44077"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "3d1bb490dc5497727353f476a49f17ca"
server_key_iv: "847f3d78695ba42fcb0b1a9b04fa1cb3"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.348685 19117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:03.362767 19117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:03.363452 19117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:03.363641 19117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:03.364387 19117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:03.365428 19117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:03.365499 19117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.365562 19117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:03.365592 19117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.374521 19117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46611
I20260927 02:48:03.374545 19244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46611 every 8 connection(s)
I20260927 02:48:03.375420 19117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:48:03.378683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 19117
I20260927 02:48:03.378813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:03.379119 392 external_mini_cluster.cc:1428] Setting key 17319ebaf67ebd585979de5c8eb53de0
I20260927 02:48:03.381167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41893
--builtin_ntp_servers=127.0.98.20:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:48:03.385262 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.377186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60481 (local address 127.0.98.62:41893)
0927 02:48:03.377332 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.377336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.378408 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 02:48:03.380068 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.380072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.380121 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.380199 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:03.381211 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.381822 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.382966 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.383323 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.383434 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.384791 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:03.384794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:03.384797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:03.384810 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:03.384834 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:03.384842 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:03.384901 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.385101 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.385153 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:48:03.386595 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.377590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41893 (local address 127.0.98.1:60481)
0927 02:48:03.378259 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.378298 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:03.379822 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:03.380404 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:03.380415 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:03.380548 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:03.381075 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.381087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.381951 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.381954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.382842 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.382851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.383478 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.383486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.383724 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.384447 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:03.384452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:03.384514 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:03.384595 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:03.384935 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:03.384947 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:03.384955 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 02:48:03.385101 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 02:48:03.385430 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
I20260927 02:48:03.387969 19245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41893
I20260927 02:48:03.388212 19245 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:03.388720 19245 heartbeater.cc:511] Master 127.0.98.62:41893 requested a full tablet report, sending...
I20260927 02:48:03.390206 19057 ts_manager.cc:194] Registered new tserver with Master: 01286026ca154525a4f825864eb44077 (127.0.98.1:46611)
I20260927 02:48:03.391328 19057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60481
W20260927 02:48:03.490187 19248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:03.490525 19248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:03.490564 19248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:03.494544 19248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:03.494596 19248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:03.494647 19248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:03.499619 19248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:03.500614 19248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:03.501971 19248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:03.511066 19253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:03.511068 19254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:03.511066 19256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:03.511318 19248 server_base.cc:1034] running on GCE node
I20260927 02:48:03.511762 19248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:03.512395 19248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:03.513576 19248 hybrid_clock.cc:648] HybridClock initialized: now 1790477283513547 us; error 47 us; skew 500 ppm
I20260927 02:48:03.515278 19248 webserver.cc:460] Webserver started at http://127.0.98.2:44665/ using document root <none> and password file <none>
I20260927 02:48:03.515822 19248 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:03.515936 19248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:03.516158 19248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:03.518069 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "94a8c932c8194ae6bb2758b71c4f0001"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "f0353e97980c336cfdf8c9b2ea6f04c1"
server_key_iv: "e6ed81751cc3e2430354517d1cc304ba"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.518607 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "94a8c932c8194ae6bb2758b71c4f0001"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "f0353e97980c336cfdf8c9b2ea6f04c1"
server_key_iv: "e6ed81751cc3e2430354517d1cc304ba"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.522159 19248 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 02:48:03.524416 19262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.525555 19248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:03.525681 19248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "94a8c932c8194ae6bb2758b71c4f0001"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "f0353e97980c336cfdf8c9b2ea6f04c1"
server_key_iv: "e6ed81751cc3e2430354517d1cc304ba"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.525784 19248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:03.537565 19248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:03.538153 19248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:03.538337 19248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:03.538862 19248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:03.539701 19248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:03.539770 19248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.539830 19248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:03.539873 19248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.548552 19248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45269
I20260927 02:48:03.548619 19375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45269 every 8 connection(s)
I20260927 02:48:03.549405 19248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:03.557298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 19248
I20260927 02:48:03.557428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:03.557729 392 external_mini_cluster.cc:1428] Setting key da1f14bdb2261946d7d2e398c0452eeb
I20260927 02:48:03.558398 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.551162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54103 (local address 127.0.98.62:41893)
0927 02:48:03.551298 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.551301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.552433 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 02:48:03.553819 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.553821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.553866 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.553939 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:03.554757 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.555275 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.556426 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.556638 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.556711 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.557902 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:03.557911 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:03.557913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:03.557923 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:03.557940 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:03.557948 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:03.558024 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.558187 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.558249 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:48:03.559170 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.551573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41893 (local address 127.0.98.2:54103)
0927 02:48:03.552260 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.552299 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:03.553629 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:03.554092 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:03.554099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:03.554174 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:03.554639 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.554650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.555390 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.555393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.556309 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.556318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.556761 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.556764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.556926 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.557612 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:03.557614 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:03.557650 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:03.557706 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:03.558038 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:03.558047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:03.558052 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:48:03.558174 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 02:48:03.558419 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260927 02:48:03.559815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41893
--builtin_ntp_servers=127.0.98.20:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:48:03.560288 19376 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41893
I20260927 02:48:03.560524 19376 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:03.561086 19376 heartbeater.cc:511] Master 127.0.98.62:41893 requested a full tablet report, sending...
I20260927 02:48:03.562127 19057 ts_manager.cc:194] Registered new tserver with Master: 94a8c932c8194ae6bb2758b71c4f0001 (127.0.98.2:45269)
I20260927 02:48:03.562958 19057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54103
W20260927 02:48:03.669162 19379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:03.669402 19379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:03.669437 19379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:03.673309 19379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:03.673354 19379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:03.673404 19379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:03.678005 19379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:03.678944 19379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:03.680286 19379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:03.689332 19384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:03.689353 19387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:03.689342 19385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:03.689754 19379 server_base.cc:1034] running on GCE node
I20260927 02:48:03.690114 19379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:03.690655 19379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:03.691851 19379 hybrid_clock.cc:648] HybridClock initialized: now 1790477283691817 us; error 25 us; skew 500 ppm
I20260927 02:48:03.693650 19379 webserver.cc:460] Webserver started at http://127.0.98.3:45983/ using document root <none> and password file <none>
I20260927 02:48:03.694216 19379 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:03.694281 19379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:03.694437 19379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:03.696435 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "fdacab8b45664fab9d20719464e7859d"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "18bba0de120ecc2e340f93cd3eb5fadd"
server_key_iv: "9e8eebc9e7a0323fcedc5c6aa32c3767"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.697090 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fdacab8b45664fab9d20719464e7859d"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "18bba0de120ecc2e340f93cd3eb5fadd"
server_key_iv: "9e8eebc9e7a0323fcedc5c6aa32c3767"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.700619 19379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:48:03.702993 19393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.704276 19379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:03.704399 19379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "fdacab8b45664fab9d20719464e7859d"
format_stamp: "Formatted at 2026-09-27 02:48:03 on dist-test-slave-gkrl"
server_key: "18bba0de120ecc2e340f93cd3eb5fadd"
server_key_iv: "9e8eebc9e7a0323fcedc5c6aa32c3767"
server_key_version: "encryptionkey@0"
I20260927 02:48:03.704497 19379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:03.719962 19379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:03.720613 19379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:03.720793 19379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:03.721300 19379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:03.722123 19379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:03.722168 19379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.722208 19379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:03.722222 19379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:03.731205 19379 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42413
I20260927 02:48:03.731225 19506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42413 every 8 connection(s)
I20260927 02:48:03.732075 19379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:03.736289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 19379
I20260927 02:48:03.736429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:03.736703 392 external_mini_cluster.cc:1428] Setting key 32918af43824e6041e25b9e7149fd0f7
I20260927 02:48:03.741353 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.733829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32865 (local address 127.0.98.62:41893)
0927 02:48:03.733974 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.733977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.734987 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 02:48:03.736410 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.736414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.736468 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.736547 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:03.737594 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.738256 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.739618 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.739849 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.739989 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.740962 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:03.740964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:03.740966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:03.740977 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:03.740994 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:03.741007 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:03.741065 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.741199 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.741246 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:48:03.742161 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.734183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41893 (local address 127.0.98.3:32865)
0927 02:48:03.734833 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.734872 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:03.736199 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:03.736815 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:03.736823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:03.736925 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:03.737484 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.737500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.738385 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.738388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.739486 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.739496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.740041 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.740044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.740176 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.740727 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:03.740728 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:03.740773 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:03.740832 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:03.741079 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:03.741085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:03.741088 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:48:03.741183 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:48:03.741382 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260927 02:48:03.743211 19507 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41893
I20260927 02:48:03.743441 19507 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:03.744002 19507 heartbeater.cc:511] Master 127.0.98.62:41893 requested a full tablet report, sending...
I20260927 02:48:03.744997 19057 ts_manager.cc:194] Registered new tserver with Master: fdacab8b45664fab9d20719464e7859d (127.0.98.3:42413)
I20260927 02:48:03.745555 19057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:32865
I20260927 02:48:03.750535 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:48:03.756388 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.752548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.98.62:41893)
0927 02:48:03.752681 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.752686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.752850 (+ 164us) server_negotiation.cc:407] Connection header received
0927 02:48:03.753028 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.753030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.753066 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.753148 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:03.753435 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.753915 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.754749 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.754964 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.755071 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.755844 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:03.755848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:03.755853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:03.755870 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:03.755929 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:03.755939 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:03.756078 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.756223 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.756257 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:48:03.758993 19057 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:48:03.761873 19057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:03.775825 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.770167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45269 (local address 127.0.0.1:36694)
0927 02:48:03.771093 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.771105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:03.771197 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:03.772044 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:03.772047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:03.772069 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:03.772156 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.772162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.773473 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.773476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.774551 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.774559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.775320 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.775324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.775424 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.775459 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:48:03.775553 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:48:03.775659 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":7,"thread_start_us":86,"threads_started":1}
I20260927 02:48:03.775825 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.770644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.0.98.2:45269)
0927 02:48:03.770956 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.770969 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.771526 (+ 557us) server_negotiation.cc:407] Connection header received
0927 02:48:03.771602 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.771607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.771748 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.771901 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:03.772284 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.773362 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.774664 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.775189 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.775320 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.775519 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.775589 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.775674 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":127,"threads_started":1}
I20260927 02:48:03.776360 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.770815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.0.98.3:42413)
0927 02:48:03.771167 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.771176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.771431 (+ 255us) server_negotiation.cc:407] Connection header received
0927 02:48:03.771518 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.771523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.771706 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.771901 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:03.772099 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.772872 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.774698 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.775464 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.775566 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.775734 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.776109 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.776201 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":86,"threads_started":1}
I20260927 02:48:03.776036 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.769988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46611 (local address 127.0.0.1:37108)
0927 02:48:03.770442 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.770489 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:03.770609 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:03.771565 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:03.771572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:03.771594 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:03.771750 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.771760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.773395 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.773399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.774593 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.774603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.775606 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.775610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.775705 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.775727 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:48:03.775778 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:48:03.775847 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":114,"thread_start_us":82,"threads_started":1}
I20260927 02:48:03.776240 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.770665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42413 (local address 127.0.0.1:56058)
0927 02:48:03.771312 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.771325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:03.771420 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:03.771857 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:03.771862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:03.771903 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:03.771987 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.771993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.772993 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.772999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.774556 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.774567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.775821 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.775824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.775940 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.775961 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:48:03.776066 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 02:48:03.776136 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":39,"thread_start_us":108,"threads_started":1}
I20260927 02:48:03.776651 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.770376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.0.98.1:46611)
0927 02:48:03.771043 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.771060 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.771086 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:48:03.771164 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.771170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.771396 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.771949 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:03.772091 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.773264 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.774726 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.775464 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.776134 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.776307 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.776362 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.776481 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":109,"threads_started":1}
I20260927 02:48:03.779027 19441 tablet_service.cc:1467] Processing CreateTablet for tablet 69a50c2ac8764fb18cf2d7615883c75b (DEFAULT_TABLE table=test-table [id=b04eb8c2290745fcbca1c81668e81663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:03.779030 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 69a50c2ac8764fb18cf2d7615883c75b (DEFAULT_TABLE table=test-table [id=b04eb8c2290745fcbca1c81668e81663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:03.779891 19441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a50c2ac8764fb18cf2d7615883c75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:03.779935 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a50c2ac8764fb18cf2d7615883c75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:03.781729 19179 tablet_service.cc:1467] Processing CreateTablet for tablet 69a50c2ac8764fb18cf2d7615883c75b (DEFAULT_TABLE table=test-table [id=b04eb8c2290745fcbca1c81668e81663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:03.782630 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a50c2ac8764fb18cf2d7615883c75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:03.786697 19525 tablet_bootstrap.cc:492] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: Bootstrap starting.
I20260927 02:48:03.787235 19524 tablet_bootstrap.cc:492] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Bootstrap starting.
I20260927 02:48:03.788847 19526 tablet_bootstrap.cc:492] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077: Bootstrap starting.
I20260927 02:48:03.788940 19525 tablet_bootstrap.cc:654] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:03.789278 19524 tablet_bootstrap.cc:654] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:03.789938 19525 log.cc:824] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:03.790033 19524 log.cc:824] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:03.791003 19526 tablet_bootstrap.cc:654] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:03.791787 19526 log.cc:824] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:03.791851 19525 tablet_bootstrap.cc:492] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: No bootstrap required, opened a new log
I20260927 02:48:03.791913 19524 tablet_bootstrap.cc:492] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: No bootstrap required, opened a new log
I20260927 02:48:03.792069 19525 ts_tablet_manager.cc:1397] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 02:48:03.792081 19524 ts_tablet_manager.cc:1397] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:48:03.793627 19526 tablet_bootstrap.cc:492] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077: No bootstrap required, opened a new log
I20260927 02:48:03.793788 19526 ts_tablet_manager.cc:1397] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:48:03.795083 19525 raft_consensus.cc:348] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.795279 19525 raft_consensus.cc:374] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:03.795310 19525 raft_consensus.cc:729] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdacab8b45664fab9d20719464e7859d, State: Initialized, Role: FOLLOWER
I20260927 02:48:03.795724 19525 consensus_queue.cc:260] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.796154 19524 raft_consensus.cc:348] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.796603 19507 heartbeater.cc:503] Master 127.0.98.62:41893 was elected leader, sending a full tablet report...
I20260927 02:48:03.796731 19525 ts_tablet_manager.cc:1428] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260927 02:48:03.796458 19524 raft_consensus.cc:374] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:03.797062 19524 raft_consensus.cc:729] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a8c932c8194ae6bb2758b71c4f0001, State: Initialized, Role: FOLLOWER
I20260927 02:48:03.797518 19524 consensus_queue.cc:260] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.797926 19526 raft_consensus.cc:348] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.798139 19376 heartbeater.cc:503] Master 127.0.98.62:41893 was elected leader, sending a full tablet report...
I20260927 02:48:03.798204 19526 raft_consensus.cc:374] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:03.798250 19526 raft_consensus.cc:729] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01286026ca154525a4f825864eb44077, State: Initialized, Role: FOLLOWER
I20260927 02:48:03.798441 19524 ts_tablet_manager.cc:1428] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:48:03.798629 19526 consensus_queue.cc:260] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.799197 19245 heartbeater.cc:503] Master 127.0.98.62:41893 was elected leader, sending a full tablet report...
I20260927 02:48:03.799491 19526 ts_tablet_manager.cc:1428] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:48:03.800112 19531 raft_consensus.cc:482] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 02:48:03.800257 19377 tablet.cc:2369] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:03.800257 19531 raft_consensus.cc:504] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.801362 19531 leader_election.cc:288] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01286026ca154525a4f825864eb44077 (127.0.98.1:46611), fdacab8b45664fab9d20719464e7859d (127.0.98.3:42413)
I20260927 02:48:03.805766 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.801656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43511 (local address 127.0.98.1:46611)
0927 02:48:03.801767 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.801772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.802205 (+ 433us) server_negotiation.cc:407] Connection header received
0927 02:48:03.802249 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.802252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.802301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.802369 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:03.802791 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.803572 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.804790 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.805240 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.805360 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.805427 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.805589 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.805650 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":9472}
I20260927 02:48:03.805766 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.801588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46611 (local address 127.0.98.2:43511)
0927 02:48:03.801881 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.801896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:03.801984 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:03.802507 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:03.802510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:03.802529 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:03.802662 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.802671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.803687 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.803690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.804676 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.804684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.805361 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.805364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.805469 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.805487 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:48:03.805575 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 02:48:03.805642 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":4}
I20260927 02:48:03.805852 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.801898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57031 (local address 127.0.98.3:42413)
0927 02:48:03.802020 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.802024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.802202 (+ 178us) server_negotiation.cc:407] Connection header received
0927 02:48:03.802375 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.802378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.802419 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.802462 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:03.802848 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.803561 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.804925 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.805300 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.805418 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.805469 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.805653 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.805714 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:48:03.805917 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.801810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42413 (local address 127.0.98.2:57031)
0927 02:48:03.802126 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.802139 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:03.802262 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:03.802579 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:03.802581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:03.802593 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:03.802734 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.802742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.803704 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.803707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.804819 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:03.804828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.805428 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.805431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:03.805525 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.805545 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:48:03.805640 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:48:03.805703 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":46,"thread_start_us":115,"threads_started":1}
I20260927 02:48:03.806366 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a50c2ac8764fb18cf2d7615883c75b" candidate_uuid: "94a8c932c8194ae6bb2758b71c4f0001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01286026ca154525a4f825864eb44077" is_pre_election: true
I20260927 02:48:03.806433 19461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a50c2ac8764fb18cf2d7615883c75b" candidate_uuid: "94a8c932c8194ae6bb2758b71c4f0001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdacab8b45664fab9d20719464e7859d" is_pre_election: true
I20260927 02:48:03.806627 19461 raft_consensus.cc:2454] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94a8c932c8194ae6bb2758b71c4f0001 in term 0.
I20260927 02:48:03.806627 19199 raft_consensus.cc:2454] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94a8c932c8194ae6bb2758b71c4f0001 in term 0.
I20260927 02:48:03.807055 19265 leader_election.cc:302] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94a8c932c8194ae6bb2758b71c4f0001, fdacab8b45664fab9d20719464e7859d; no voters:
I20260927 02:48:03.807271 19531 raft_consensus.cc:2781] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:03.807348 19531 raft_consensus.cc:482] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:03.807389 19531 raft_consensus.cc:3037] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:03.808343 19531 raft_consensus.cc:504] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.808629 19531 leader_election.cc:288] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [CANDIDATE]: Term 1 election: Requested vote from peers 01286026ca154525a4f825864eb44077 (127.0.98.1:46611), fdacab8b45664fab9d20719464e7859d (127.0.98.3:42413)
I20260927 02:48:03.808920 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a50c2ac8764fb18cf2d7615883c75b" candidate_uuid: "94a8c932c8194ae6bb2758b71c4f0001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01286026ca154525a4f825864eb44077"
I20260927 02:48:03.809026 19199 raft_consensus.cc:3037] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:03.809029 19461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a50c2ac8764fb18cf2d7615883c75b" candidate_uuid: "94a8c932c8194ae6bb2758b71c4f0001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdacab8b45664fab9d20719464e7859d"
I20260927 02:48:03.809157 19461 raft_consensus.cc:3037] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:03.809986 19199 raft_consensus.cc:2454] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94a8c932c8194ae6bb2758b71c4f0001 in term 1.
I20260927 02:48:03.810148 19461 raft_consensus.cc:2454] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94a8c932c8194ae6bb2758b71c4f0001 in term 1.
I20260927 02:48:03.810271 19265 leader_election.cc:302] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01286026ca154525a4f825864eb44077, 94a8c932c8194ae6bb2758b71c4f0001; no voters:
I20260927 02:48:03.810565 19531 raft_consensus.cc:2781] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:03.810781 19531 raft_consensus.cc:686] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 1 LEADER]: Becoming Leader. State: Replica: 94a8c932c8194ae6bb2758b71c4f0001, State: Running, Role: LEADER
I20260927 02:48:03.811066 19531 consensus_queue.cc:237] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } }
I20260927 02:48:03.813964 19056 catalog_manager.cc:5495] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94a8c932c8194ae6bb2758b71c4f0001 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "94a8c932c8194ae6bb2758b71c4f0001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94a8c932c8194ae6bb2758b71c4f0001" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 } health_report { overall_health: UNKNOWN } } }
I20260927 02:48:03.829068 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.825771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.0.98.2:45269)
0927 02:48:03.825915 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:03.825918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:03.825931 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:48:03.826031 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:03.826034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:03.826079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:03.826165 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:03.826360 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.826845 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.827651 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:03.827860 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:03.827998 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:03.828263 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:03.828636 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:03.828737 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:03.828885 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:48:03.828914 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 02:48:03.834378 19461 raft_consensus.cc:1261] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 1 FOLLOWER]: Refusing update from remote peer 94a8c932c8194ae6bb2758b71c4f0001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:03.834422 19199 raft_consensus.cc:1261] T 69a50c2ac8764fb18cf2d7615883c75b P 01286026ca154525a4f825864eb44077 [term 1 FOLLOWER]: Refusing update from remote peer 94a8c932c8194ae6bb2758b71c4f0001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:03.835103 19534 consensus_queue.cc:1035] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdacab8b45664fab9d20719464e7859d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:03.835237 19531 consensus_queue.cc:1035] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01286026ca154525a4f825864eb44077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:03.842777 19538 mvcc.cc:205] Tried to move back new op lower bound from 7333794954579193856 to 7333794954492313600. Current Snapshot: MvccSnapshot[applied={T|T < 7333794954579193856}]
I20260927 02:48:03.844669 19539 mvcc.cc:205] Tried to move back new op lower bound from 7333794954579193856 to 7333794954492313600. Current Snapshot: MvccSnapshot[applied={T|T < 7333794954579193856}]
I20260927 02:48:03.851468 19057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:03.851707 19057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:03.854126 19057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c7f943579c947f484c234e4bbda8c1b: Sending DeleteTablet for 3 replicas of tablet 69a50c2ac8764fb18cf2d7615883c75b
I20260927 02:48:03.854902 19310 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a50c2ac8764fb18cf2d7615883c75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:03 UTC) from {username='slave'} at 127.0.0.1:36694
I20260927 02:48:03.854902 19441 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a50c2ac8764fb18cf2d7615883c75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:03 UTC) from {username='slave'} at 127.0.0.1:56058
I20260927 02:48:03.855476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 19117
I20260927 02:48:03.855484 19179 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a50c2ac8764fb18cf2d7615883c75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:03 UTC) from {username='slave'} at 127.0.0.1:37108
I20260927 02:48:03.855574 19548 tablet_replica.cc:330] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: stopping tablet replica
I20260927 02:48:03.855857 19548 raft_consensus.cc:2229] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:03.856297 19548 raft_consensus.cc:2258] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:03.857788 19548 ts_tablet_manager.cc:1905] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:03.860617 19548 ts_tablet_manager.cc:1918] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:03.860764 19548 log.cc:1196] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/69a50c2ac8764fb18cf2d7615883c75b
I20260927 02:48:03.861160 19548 ts_tablet_manager.cc:1939] T 69a50c2ac8764fb18cf2d7615883c75b P 94a8c932c8194ae6bb2758b71c4f0001: Deleting consensus metadata
W20260927 02:48:03.862177 19043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46611 (error 108)
I20260927 02:48:03.862274 19547 tablet_replica.cc:330] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: stopping tablet replica
I20260927 02:48:03.862293 19041 catalog_manager.cc:4841] TS 94a8c932c8194ae6bb2758b71c4f0001 (127.0.98.2:45269): tablet 69a50c2ac8764fb18cf2d7615883c75b (table test-table [id=b04eb8c2290745fcbca1c81668e81663]) successfully deleted
I20260927 02:48:03.862561 19547 raft_consensus.cc:2229] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:03.862918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 19248
I20260927 02:48:03.864409 19547 raft_consensus.cc:2258] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:03.865392 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.865068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46611 (local address 127.0.0.1:37114)
0927 02:48:03.865193 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.865304 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:48:03.865533 19043 catalog_manager.cc:4584] TS 01286026ca154525a4f825864eb44077 (127.0.98.1:46611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69a50c2ac8764fb18cf2d7615883c75b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46611: connect: Connection refused (error 111)
I20260927 02:48:03.865835 19547 ts_tablet_manager.cc:1905] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:03.867766 19547 ts_tablet_manager.cc:1918] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:03.867843 19547 log.cc:1196] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/69a50c2ac8764fb18cf2d7615883c75b
I20260927 02:48:03.868155 19547 ts_tablet_manager.cc:1939] T 69a50c2ac8764fb18cf2d7615883c75b P fdacab8b45664fab9d20719464e7859d: Deleting consensus metadata
I20260927 02:48:03.869127 19043 catalog_manager.cc:4841] TS fdacab8b45664fab9d20719464e7859d (127.0.98.3:42413): tablet 69a50c2ac8764fb18cf2d7615883c75b (table test-table [id=b04eb8c2290745fcbca1c81668e81663]) successfully deleted
I20260927 02:48:03.869648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 19379
I20260927 02:48:03.875010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 19026
2026-09-27T02:48:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (932 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:03 dist-test-slave-gkrl krb5kdc[19552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:03 dist-test-slave-gkrl krb5kdc[19552](info): set up 2 sockets
Sep 27 02:48:03 dist-test-slave-gkrl krb5kdc[19552](info): commencing operation
krb5kdc: starting...
W20260927 02:48:04.904311 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:04 dist-test-slave-gkrl krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:05.023528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40787
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:05.129240 19565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:05.129540 19565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:05.129619 19565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:05.133599 19565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:05.133687 19565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:05.133728 19565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:05.133790 19565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:05.133831 19565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:05.138921 19565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:05.139925 19565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:05.140898 19565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:05.148770 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:05.148823 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:05.148898 19565 server_base.cc:1034] running on GCE node
W20260927 02:48:05.149015 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:05.149616 19565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:05.150636 19565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:05.151823 19565 hybrid_clock.cc:648] HybridClock initialized: now 1790477285151806 us; error 40 us; skew 500 ppm
Sep 27 02:48:05 dist-test-slave-gkrl krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:05.154543 19565 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:05.155321 19565 webserver.cc:460] Webserver started at http://127.0.98.62:33361/ using document root <none> and password file <none>
I20260927 02:48:05.155797 19565 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:05.155853 19565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:05.156088 19565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:05.157934 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "c4c774bcb81f485e996dbff7ddbcad29"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "0b0c54a9c71ed58906096b4d27bd3d91"
server_key_iv: "1a88527840f7998ff81962951c29b1b1"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.158475 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4c774bcb81f485e996dbff7ddbcad29"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "0b0c54a9c71ed58906096b4d27bd3d91"
server_key_iv: "1a88527840f7998ff81962951c29b1b1"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.161922 19565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:48:05.164119 19580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.165387 19565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:05.165486 19565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "c4c774bcb81f485e996dbff7ddbcad29"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "0b0c54a9c71ed58906096b4d27bd3d91"
server_key_iv: "1a88527840f7998ff81962951c29b1b1"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.165570 19565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:05.185529 19565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:05.188426 19565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:05.188567 19565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:05.195590 19565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40787
I20260927 02:48:05.195616 19632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40787 every 8 connection(s)
I20260927 02:48:05.196578 19565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:05.199024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 19565
I20260927 02:48:05.199162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:05.199324 19633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:05.199479 392 external_mini_cluster.cc:1428] Setting key 21267e83ed34ffa32c2341670d9717bb
I20260927 02:48:05.206327 19633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29: Bootstrap starting.
I20260927 02:48:05.208580 19633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29: Neither blocks nor log segments found. Creating new log.
Sep 27 02:48:05 dist-test-slave-gkrl krb5kdc[19552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:05.209383 19633 log.cc:824] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:05.211621 19633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29: No bootstrap required, opened a new log
I20260927 02:48:05.214488 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.200959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.0.98.62:40787)
0927 02:48:05.201391 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:05.201408 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:05.201452 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:48:05.202115 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:05.202136 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:05.202542 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:05.202908 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:05.203540 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.205053 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.205904 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.206594 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.206746 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:05.209315 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:05.209321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:05.209337 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:05.209365 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:05.211438 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.211929 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.211935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.211942 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.212085 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.212342 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.212345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.212347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.212826 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:05.213011 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:05.213211 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:48:05.213490 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260927 02:48:05.216044 19633 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c774bcb81f485e996dbff7ddbcad29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } }
I20260927 02:48:05.216306 19633 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:05.216398 19633 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c774bcb81f485e996dbff7ddbcad29, State: Initialized, Role: FOLLOWER
I20260927 02:48:05.216841 19633 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c774bcb81f485e996dbff7ddbcad29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } }
I20260927 02:48:05.216985 19633 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:05.217049 19633 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:05.217140 19633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:05.217964 19633 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c774bcb81f485e996dbff7ddbcad29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } }
I20260927 02:48:05.218289 19633 leader_election.cc:302] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4c774bcb81f485e996dbff7ddbcad29; no voters:
I20260927 02:48:05.218566 19633 leader_election.cc:288] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:05.218707 19638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:05.218937 19638 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [term 1 LEADER]: Becoming Leader. State: Replica: c4c774bcb81f485e996dbff7ddbcad29, State: Running, Role: LEADER
I20260927 02:48:05.219244 19638 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c774bcb81f485e996dbff7ddbcad29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } }
I20260927 02:48:05.219732 19633 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:05.220999 19639 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4c774bcb81f485e996dbff7ddbcad29. Latest consensus state: current_term: 1 leader_uuid: "c4c774bcb81f485e996dbff7ddbcad29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c774bcb81f485e996dbff7ddbcad29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } } }
I20260927 02:48:05.221128 19639 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:05.220971 19640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4c774bcb81f485e996dbff7ddbcad29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c774bcb81f485e996dbff7ddbcad29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } } }
I20260927 02:48:05.221354 19640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29 [sys.catalog]: This master's current role is: LEADER
W20260927 02:48:05.224438 19653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:05.224499 19653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:05.224574 19648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:05.225240 19648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:05.229988 19648 catalog_manager.cc:1347] Generated new cluster ID: ce91ff3a80bf45a6b30042e25e48291f
I20260927 02:48:05.230046 19648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:05.252323 19648 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:05.253599 19648 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:05.264065 19648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29: Generated new TSK 0
I20260927 02:48:05.264907 19648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:48:05.295630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:05.403786 19659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:05.404080 19659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:05.404152 19659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:05.408241 19659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:05.408316 19659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:05.408406 19659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:05.413529 19659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:05.414518 19659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:05.415479 19659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:05.425063 19664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:05.425060 19667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:05.425052 19665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:05.425256 19659 server_base.cc:1034] running on GCE node
I20260927 02:48:05.425709 19659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:05.426329 19659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:05.427533 19659 hybrid_clock.cc:648] HybridClock initialized: now 1790477285427524 us; error 30 us; skew 500 ppm
Sep 27 02:48:05 dist-test-slave-gkrl krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:05.430404 19659 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:48:05.431242 19659 webserver.cc:460] Webserver started at http://127.0.98.1:40573/ using document root <none> and password file <none>
I20260927 02:48:05.431741 19659 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:05.431797 19659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:05.432024 19659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:05.433964 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "06ae67ffe8f64168a6eb29a204b04d8a"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "c035e2f5d3bc12d20d90e1a32ea73aa8"
server_key_iv: "262a2ad33bc003dc352cfa4508dac3bf"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.434470 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "06ae67ffe8f64168a6eb29a204b04d8a"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "c035e2f5d3bc12d20d90e1a32ea73aa8"
server_key_iv: "262a2ad33bc003dc352cfa4508dac3bf"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.438457 19659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 02:48:05.440990 19674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.442317 19659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:48:05.442420 19659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "06ae67ffe8f64168a6eb29a204b04d8a"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "c035e2f5d3bc12d20d90e1a32ea73aa8"
server_key_iv: "262a2ad33bc003dc352cfa4508dac3bf"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.442507 19659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:05.461792 19659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:05.465224 19659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:05.465411 19659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:05.465987 19659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:05.466764 19659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:05.466812 19659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.466851 19659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:05.466874 19659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.475809 19659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46589
I20260927 02:48:05.475903 19787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46589 every 8 connection(s)
I20260927 02:48:05.476734 19659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:48:05.481747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 19659
I20260927 02:48:05.481885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:05.482165 392 external_mini_cluster.cc:1428] Setting key ea1fc8dff99638f827bacb89048d1082
Sep 27 02:48:05 dist-test-slave-gkrl krb5kdc[19552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:05.490949 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.478486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56821 (local address 127.0.98.62:40787)
0927 02:48:05.478614 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:05.478618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:05.479414 (+ 796us) server_negotiation.cc:407] Connection header received
0927 02:48:05.480380 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:05.480384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:05.480443 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:05.480547 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:05.481754 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.482441 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.483946 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.484182 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.484266 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:05.486926 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:05.486932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:05.486938 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:05.486971 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:05.488829 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.489313 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.489320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.489323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.489422 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.489751 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.489754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.489757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.490006 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:05.490109 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:05.490604 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 02:48:05.490743 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:48:05.491653 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.478808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40787 (local address 127.0.98.1:56821)
0927 02:48:05.479255 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:05.479304 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:05.480160 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:05.480720 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:05.480727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:05.481102 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:05.481592 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:05.481606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.482584 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.482587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:05.483774 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:05.483786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.484352 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.484359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:05.484606 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:05.485250 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:05.485278 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:05.486756 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:05.488949 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:05.488954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:05.488965 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:05.489211 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:05.489524 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:05.489526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:05.489528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:05.489655 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:05.490107 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:05.490111 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:05.490344 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 02:48:05.490512 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 02:48:05.490768 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:48:05.492954 19788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40787
I20260927 02:48:05.493225 19788 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:05.493695 19788 heartbeater.cc:511] Master 127.0.98.62:40787 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:48:05.495334 19596 ts_manager.cc:194] Registered new tserver with Master: 06ae67ffe8f64168a6eb29a204b04d8a (127.0.98.1:46589)
I20260927 02:48:05.496627 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:48:05.510828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:05.618848 19793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:05.619092 19793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:05.619135 19793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:05.623472 19793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:05.623524 19793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:05.623607 19793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:05.628744 19793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:05.629714 19793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:05.630715 19793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:05.640132 19801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:05.640111 19799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:05.640097 19798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:05.640364 19793 server_base.cc:1034] running on GCE node
I20260927 02:48:05.640779 19793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:05.641410 19793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:05.642642 19793 hybrid_clock.cc:648] HybridClock initialized: now 1790477285642627 us; error 27 us; skew 500 ppm
Sep 27 02:48:05 dist-test-slave-gkrl krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:05.645499 19793 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:48:05.646476 19793 webserver.cc:460] Webserver started at http://127.0.98.2:37127/ using document root <none> and password file <none>
I20260927 02:48:05.646977 19793 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:05.647034 19793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:05.647187 19793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:05.649178 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "953250779d3e4f539b48643fcbe290c9"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "b377c7651df3ff93991d0e2ffc6c55a4"
server_key_iv: "57d88b0267329ee0746185bc54dab42d"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.649688 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "953250779d3e4f539b48643fcbe290c9"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "b377c7651df3ff93991d0e2ffc6c55a4"
server_key_iv: "57d88b0267329ee0746185bc54dab42d"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.653239 19793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:48:05.655522 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.656759 19793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:05.656850 19793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "953250779d3e4f539b48643fcbe290c9"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "b377c7651df3ff93991d0e2ffc6c55a4"
server_key_iv: "57d88b0267329ee0746185bc54dab42d"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.656935 19793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:05.723769 19793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:05.727388 19793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:05.727593 19793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:05.728274 19793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:05.729059 19793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:05.729107 19793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.729146 19793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:05.729161 19793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.738226 19793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46321
I20260927 02:48:05.738253 19921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46321 every 8 connection(s)
I20260927 02:48:05.739113 19793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:05.747604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 19793
I20260927 02:48:05.747761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:05.748104 392 external_mini_cluster.cc:1428] Setting key 995ded4f37d9d5b9b3372405d6467f8e
Sep 27 02:48:05 dist-test-slave-gkrl krb5kdc[19552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:05.753572 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.741025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58733 (local address 127.0.98.62:40787)
0927 02:48:05.741155 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:05.741160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:05.742191 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 02:48:05.743139 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:05.743142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:05.743202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:05.743298 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:05.744540 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.745075 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.746429 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.746652 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.746762 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:05.749465 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:05.749475 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:05.749478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:05.749522 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:05.751230 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.751949 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.751952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.751953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.752029 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.752460 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.752462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.752463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.752628 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:05.752701 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:05.753354 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 02:48:05.753455 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:48:05.754827 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.741406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40787 (local address 127.0.98.2:58733)
0927 02:48:05.742050 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:05.742089 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:05.742928 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:05.743476 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:05.743483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:05.743842 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:05.744378 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:05.744414 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.745218 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.745221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:05.746297 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:05.746307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.746806 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.746810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:05.746977 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:05.747606 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:05.747621 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:05.749285 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:05.751390 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:05.751399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:05.751418 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:05.751805 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:05.752124 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:05.752130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:05.752133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:05.752356 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:05.752733 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:05.752740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:05.753080 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 02:48:05.753293 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 02:48:05.753672 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":112,"threads_started":1}
I20260927 02:48:05.756294 19922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40787
I20260927 02:48:05.756562 19922 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:05.757154 19922 heartbeater.cc:511] Master 127.0.98.62:40787 requested a full tablet report, sending...
I20260927 02:48:05.758392 19596 ts_manager.cc:194] Registered new tserver with Master: 953250779d3e4f539b48643fcbe290c9 (127.0.98.2:46321)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:48:05.759189 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:48:05.775329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:05.887116 19927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:05.887396 19927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:05.887441 19927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:05.891516 19927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:05.891571 19927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:05.891664 19927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:05.897992 19927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:05.899063 19927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:05.900180 19927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:05.909651 19932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:05.909812 19935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:05.909708 19933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:05.909730 19927 server_base.cc:1034] running on GCE node
I20260927 02:48:05.910272 19927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:05.910907 19927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:05.912060 19927 hybrid_clock.cc:648] HybridClock initialized: now 1790477285912028 us; error 45 us; skew 500 ppm
Sep 27 02:48:05 dist-test-slave-gkrl krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:05.914767 19927 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:48:05.915632 19927 webserver.cc:460] Webserver started at http://127.0.98.3:35673/ using document root <none> and password file <none>
I20260927 02:48:05.916246 19927 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:05.916353 19927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:05.916579 19927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:05.918534 19927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "d3502ea247e34306b25be77c54fcd741"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "4c2b425a25ec913e0e71316a5028a03a"
server_key_iv: "2572aa468fca9407b1c5a0ba58711a8d"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.919085 19927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d3502ea247e34306b25be77c54fcd741"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "4c2b425a25ec913e0e71316a5028a03a"
server_key_iv: "2572aa468fca9407b1c5a0ba58711a8d"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.922955 19927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 02:48:05.925426 19942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.926669 19927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:05.926806 19927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "d3502ea247e34306b25be77c54fcd741"
format_stamp: "Formatted at 2026-09-27 02:48:05 on dist-test-slave-gkrl"
server_key: "4c2b425a25ec913e0e71316a5028a03a"
server_key_iv: "2572aa468fca9407b1c5a0ba58711a8d"
server_key_version: "encryptionkey@0"
I20260927 02:48:05.926930 19927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:05.950510 19927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:05.953903 19927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:05.954109 19927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:05.954707 19927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:05.955513 19927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:05.955585 19927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.955654 19927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:05.955705 19927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:05.964918 19927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44401
I20260927 02:48:05.964938 20055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44401 every 8 connection(s)
I20260927 02:48:05.965869 19927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:05.971720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 19927
I20260927 02:48:05.971850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:05.972191 392 external_mini_cluster.cc:1428] Setting key 660168700fc6bb14245b1b407a028a10
Sep 27 02:48:05 dist-test-slave-gkrl krb5kdc[19552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:05.978642 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.967614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34857 (local address 127.0.98.62:40787)
0927 02:48:05.967746 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:05.967751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:05.968610 (+ 859us) server_negotiation.cc:407] Connection header received
0927 02:48:05.969563 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:05.969566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:05.969610 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:05.969684 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:05.970812 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.971336 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.972702 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.973011 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.973124 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:05.975559 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:05.975562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:05.975564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:05.975586 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:05.976880 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.977314 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.977317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.977318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.977403 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.977682 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.977684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.977685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.977938 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:05.978017 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:05.978439 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 02:48:05.978532 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:48:05.979413 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.968008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40787 (local address 127.0.98.3:34857)
0927 02:48:05.968454 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:05.968494 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:05.969369 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:05.969906 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:05.969912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:05.970249 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:05.970694 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:05.970706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.971447 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.971450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:05.972553 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:05.972566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.973165 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.973170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:05.973322 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:05.973955 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:05.973970 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:05.975420 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:05.976986 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:05.976990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:05.977000 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:05.977232 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:05.977481 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:05.977483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:05.977484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:05.977605 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:05.978027 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:05.978031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:05.978234 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 02:48:05.978381 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 02:48:05.978606 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260927 02:48:05.980530 20056 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40787
I20260927 02:48:05.980754 20056 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:05.981225 20056 heartbeater.cc:511] Master 127.0.98.62:40787 requested a full tablet report, sending...
I20260927 02:48:05.982204 19596 ts_manager.cc:194] Registered new tserver with Master: d3502ea247e34306b25be77c54fcd741 (127.0.98.3:44401)
I20260927 02:48:05.982746 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34857
I20260927 02:48:05.985800 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:48:05.994230 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.987632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.98.62:40787)
0927 02:48:05.987783 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:05.987787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:05.988020 (+ 233us) server_negotiation.cc:407] Connection header received
0927 02:48:05.988108 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:05.988110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:05.988145 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:05.988210 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:05.988828 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.989295 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.990068 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:05.990267 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:05.990341 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:05.991634 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:05.991636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:05.991638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:05.991654 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:05.992935 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.993291 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.993293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.993294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.993361 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:05.993588 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:05.993590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:05.993591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:05.993773 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:05.993872 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:05.994041 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 02:48:05.994124 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 02:48:05.996788 19596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:48:05.999678 19596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:06.015520 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.010310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.0.98.1:46589)
0927 02:48:06.010930 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.010936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.010957 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:48:06.011050 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.011057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.011236 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.011363 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:06.011766 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.012886 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.014181 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.014807 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.014942 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.015090 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.015266 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.015362 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":72,"threads_started":1}
I20260927 02:48:06.015513 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.009693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46589 (local address 127.0.0.1:39170)
0927 02:48:06.010835 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.010850 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:06.010939 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:06.011500 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:06.011503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:06.011524 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:06.011634 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.011641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.013039 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.013042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.014046 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.014054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.014939 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.014943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.015056 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.015091 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:48:06.015234 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 02:48:06.015363 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":86,"thread_start_us":103,"threads_started":1}
I20260927 02:48:06.015573 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.009069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46321 (local address 127.0.0.1:35164)
0927 02:48:06.009953 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.009966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:06.010044 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:06.011328 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:06.011331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:06.011352 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:06.011455 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.011461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.012603 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.012609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.013970 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.013985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.015117 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.015122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.015222 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.015255 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 02:48:06.015313 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:48:06.015392 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260927 02:48:06.015733 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.008938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44401 (local address 127.0.0.1:45208)
0927 02:48:06.009439 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.009494 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:06.009617 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:06.011113 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:06.011119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:06.011149 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:06.011302 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.011312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.013012 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.013016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.014285 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.014294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.015362 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.015366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.015445 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.015470 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:48:06.015518 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:48:06.015586 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":76,"thread_start_us":83,"threads_started":1}
I20260927 02:48:06.016311 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.009745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.0.98.2:46321)
0927 02:48:06.010363 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.010380 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.010407 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:48:06.010484 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.010491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.011156 (+ 665us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.011350 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:06.011562 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.012462 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.014131 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.014960 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.015779 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.015990 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.016046 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.016146 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":170,"threads_started":1}
I20260927 02:48:06.016491 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.010147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.0.98.3:44401)
0927 02:48:06.010585 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.010600 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.010635 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:48:06.010719 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.010726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.010930 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.011129 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:06.011649 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.012875 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.014429 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.015223 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.016035 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.016184 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.016232 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.016317 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":136,"threads_started":1}
I20260927 02:48:06.018853 19856 tablet_service.cc:1467] Processing CreateTablet for tablet ee19bfc2362642ae94ff2c598d60d22b (DEFAULT_TABLE table=test-table [id=5af15b0db50d439fbb8e750fcab74a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:06.018990 19722 tablet_service.cc:1467] Processing CreateTablet for tablet ee19bfc2362642ae94ff2c598d60d22b (DEFAULT_TABLE table=test-table [id=5af15b0db50d439fbb8e750fcab74a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:06.018992 19990 tablet_service.cc:1467] Processing CreateTablet for tablet ee19bfc2362642ae94ff2c598d60d22b (DEFAULT_TABLE table=test-table [id=5af15b0db50d439fbb8e750fcab74a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:06.019927 19856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee19bfc2362642ae94ff2c598d60d22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:06.020049 19722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee19bfc2362642ae94ff2c598d60d22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:06.020056 19990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee19bfc2362642ae94ff2c598d60d22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:06.026587 20073 tablet_bootstrap.cc:492] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Bootstrap starting.
I20260927 02:48:06.027761 20074 tablet_bootstrap.cc:492] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a: Bootstrap starting.
I20260927 02:48:06.028667 20073 tablet_bootstrap.cc:654] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:06.029561 20074 tablet_bootstrap.cc:654] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:06.029747 20073 log.cc:824] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:06.030402 20075 tablet_bootstrap.cc:492] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: Bootstrap starting.
I20260927 02:48:06.030447 20074 log.cc:824] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:06.032043 20073 tablet_bootstrap.cc:492] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: No bootstrap required, opened a new log
I20260927 02:48:06.032269 20073 ts_tablet_manager.cc:1397] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:48:06.032410 20074 tablet_bootstrap.cc:492] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a: No bootstrap required, opened a new log
I20260927 02:48:06.032407 20075 tablet_bootstrap.cc:654] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:06.032562 20074 ts_tablet_manager.cc:1397] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:48:06.033525 20075 log.cc:824] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:06.035508 20075 tablet_bootstrap.cc:492] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: No bootstrap required, opened a new log
I20260927 02:48:06.035678 20075 ts_tablet_manager.cc:1397] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:48:06.036228 20074 raft_consensus.cc:348] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.036252 20073 raft_consensus.cc:348] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.036463 20074 raft_consensus.cc:374] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:06.036516 20074 raft_consensus.cc:729] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ae67ffe8f64168a6eb29a204b04d8a, State: Initialized, Role: FOLLOWER
I20260927 02:48:06.036518 20073 raft_consensus.cc:374] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:06.036576 20073 raft_consensus.cc:729] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953250779d3e4f539b48643fcbe290c9, State: Initialized, Role: FOLLOWER
I20260927 02:48:06.036939 20074 consensus_queue.cc:260] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.036999 20073 consensus_queue.cc:260] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.037549 19788 heartbeater.cc:503] Master 127.0.98.62:40787 was elected leader, sending a full tablet report...
I20260927 02:48:06.037909 20073 ts_tablet_manager.cc:1428] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:48:06.038208 19922 heartbeater.cc:503] Master 127.0.98.62:40787 was elected leader, sending a full tablet report...
I20260927 02:48:06.038978 20074 ts_tablet_manager.cc:1428] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:48:06.040473 20075 raft_consensus.cc:348] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.040731 20075 raft_consensus.cc:374] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:06.040788 20075 raft_consensus.cc:729] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3502ea247e34306b25be77c54fcd741, State: Initialized, Role: FOLLOWER
I20260927 02:48:06.041179 20075 consensus_queue.cc:260] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.041878 20056 heartbeater.cc:503] Master 127.0.98.62:40787 was elected leader, sending a full tablet report...
I20260927 02:48:06.042203 20075 ts_tablet_manager.cc:1428] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:48:06.208667 20081 raft_consensus.cc:482] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:06.208847 20081 raft_consensus.cc:504] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.209957 20081 leader_election.cc:288] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 953250779d3e4f539b48643fcbe290c9 (127.0.98.2:46321), 06ae67ffe8f64168a6eb29a204b04d8a (127.0.98.1:46589)
I20260927 02:48:06.213761 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.210367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53155 (local address 127.0.98.1:46589)
0927 02:48:06.210517 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.210522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.210681 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:48:06.210832 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.210835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.210884 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.210968 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:06.211248 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.211770 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.212810 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.213226 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.213346 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.213410 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.213576 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.213653 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 02:48:06.213775 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.210246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46589 (local address 127.0.98.3:53155)
0927 02:48:06.210582 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.210593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:06.210717 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:06.210980 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:06.210982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:06.210997 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:06.211119 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.211125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.211906 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.211909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.212688 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.212697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.213347 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.213350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.213446 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.213469 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:48:06.213560 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:48:06.213629 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":115,"threads_started":1}
I20260927 02:48:06.213965 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.210154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46321 (local address 127.0.98.3:43501)
0927 02:48:06.210525 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.210541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:06.210653 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:06.210904 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:06.210906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:06.210921 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:06.211040 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.211054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.211786 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.211792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.212631 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.212639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.213539 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.213543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.213668 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.213696 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:48:06.213786 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:48:06.213850 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298}
I20260927 02:48:06.213981 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.210284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43501 (local address 127.0.98.2:46321)
0927 02:48:06.210435 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.210440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.210619 (+ 179us) server_negotiation.cc:407] Connection header received
0927 02:48:06.210748 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.210751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.210802 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.210890 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:06.211162 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.211671 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.212760 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.213384 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.213528 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.213588 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.213801 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.213878 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:48:06.214569 19742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee19bfc2362642ae94ff2c598d60d22b" candidate_uuid: "d3502ea247e34306b25be77c54fcd741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" is_pre_election: true
I20260927 02:48:06.214673 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee19bfc2362642ae94ff2c598d60d22b" candidate_uuid: "d3502ea247e34306b25be77c54fcd741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "953250779d3e4f539b48643fcbe290c9" is_pre_election: true
I20260927 02:48:06.214901 19742 raft_consensus.cc:2454] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3502ea247e34306b25be77c54fcd741 in term 0.
I20260927 02:48:06.214993 19876 raft_consensus.cc:2454] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3502ea247e34306b25be77c54fcd741 in term 0.
I20260927 02:48:06.215366 19944 leader_election.cc:302] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ae67ffe8f64168a6eb29a204b04d8a, d3502ea247e34306b25be77c54fcd741; no voters:
I20260927 02:48:06.215662 20081 raft_consensus.cc:2781] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:06.215744 20081 raft_consensus.cc:482] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:06.215786 20081 raft_consensus.cc:3037] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 0 FOLLOWER]: Advancing to term 1
W20260927 02:48:06.216745 20057 tablet.cc:2369] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:06.216913 20081 raft_consensus.cc:504] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.217267 20081 leader_election.cc:288] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [CANDIDATE]: Term 1 election: Requested vote from peers 953250779d3e4f539b48643fcbe290c9 (127.0.98.2:46321), 06ae67ffe8f64168a6eb29a204b04d8a (127.0.98.1:46589)
I20260927 02:48:06.217638 19742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee19bfc2362642ae94ff2c598d60d22b" candidate_uuid: "d3502ea247e34306b25be77c54fcd741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ae67ffe8f64168a6eb29a204b04d8a"
I20260927 02:48:06.217639 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee19bfc2362642ae94ff2c598d60d22b" candidate_uuid: "d3502ea247e34306b25be77c54fcd741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "953250779d3e4f539b48643fcbe290c9"
I20260927 02:48:06.217741 19742 raft_consensus.cc:3037] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:06.217808 19876 raft_consensus.cc:3037] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:06.218755 19742 raft_consensus.cc:2454] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3502ea247e34306b25be77c54fcd741 in term 1.
I20260927 02:48:06.218766 19876 raft_consensus.cc:2454] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3502ea247e34306b25be77c54fcd741 in term 1.
I20260927 02:48:06.219043 19944 leader_election.cc:302] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ae67ffe8f64168a6eb29a204b04d8a, d3502ea247e34306b25be77c54fcd741; no voters:
I20260927 02:48:06.219278 20081 raft_consensus.cc:2781] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:06.219506 20081 raft_consensus.cc:686] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 1 LEADER]: Becoming Leader. State: Replica: d3502ea247e34306b25be77c54fcd741, State: Running, Role: LEADER
I20260927 02:48:06.219815 20081 consensus_queue.cc:237] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } }
I20260927 02:48:06.223008 19595 catalog_manager.cc:5495] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3502ea247e34306b25be77c54fcd741 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d3502ea247e34306b25be77c54fcd741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3502ea247e34306b25be77c54fcd741" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 } health_report { overall_health: UNKNOWN } } }
W20260927 02:48:06.228060 19789 tablet.cc:2369] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:48:06.240231 19923 tablet.cc:2369] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:06.248427 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.244981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.98.3:44401)
0927 02:48:06.245122 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.245126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.245139 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:48:06.245261 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.245263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.245333 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.245418 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:06.245656 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.246165 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.247128 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.247342 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.247422 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.247659 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:06.248097 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:06.248185 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.248251 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.248312 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:48:06.253818 19876 raft_consensus.cc:1261] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 1 FOLLOWER]: Refusing update from remote peer d3502ea247e34306b25be77c54fcd741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:06.253818 19742 raft_consensus.cc:1261] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 1 FOLLOWER]: Refusing update from remote peer d3502ea247e34306b25be77c54fcd741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:06.254489 20081 consensus_queue.cc:1035] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ae67ffe8f64168a6eb29a204b04d8a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:06.254683 20083 consensus_queue.cc:1035] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "953250779d3e4f539b48643fcbe290c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:06.261461 20086 mvcc.cc:205] Tried to move back new op lower bound from 7333794964488830976 to 7333794964359049216. Current Snapshot: MvccSnapshot[applied={T|T < 7333794964488830976}]
I20260927 02:48:06.263628 20088 mvcc.cc:205] Tried to move back new op lower bound from 7333794964488830976 to 7333794964359049216. Current Snapshot: MvccSnapshot[applied={T|T < 7333794964488830976}]
I20260927 02:48:06.263916 20087 mvcc.cc:205] Tried to move back new op lower bound from 7333794964488830976 to 7333794964359049216. Current Snapshot: MvccSnapshot[applied={T|T < 7333794964488830976 or (T in {7333794964488830976})}]
I20260927 02:48:06.271209 19597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:06.271386 19597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:06.273792 19597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4c774bcb81f485e996dbff7ddbcad29: Sending DeleteTablet for 3 replicas of tablet ee19bfc2362642ae94ff2c598d60d22b
I20260927 02:48:06.274549 19856 tablet_service.cc:1514] Processing DeleteTablet for tablet ee19bfc2362642ae94ff2c598d60d22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35164
I20260927 02:48:06.274617 19722 tablet_service.cc:1514] Processing DeleteTablet for tablet ee19bfc2362642ae94ff2c598d60d22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39170
I20260927 02:48:06.274606 19990 tablet_service.cc:1514] Processing DeleteTablet for tablet ee19bfc2362642ae94ff2c598d60d22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45208
I20260927 02:48:06.275030 20097 tablet_replica.cc:330] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a: stopping tablet replica
I20260927 02:48:06.275300 20095 tablet_replica.cc:330] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: stopping tablet replica
I20260927 02:48:06.275339 20097 raft_consensus.cc:2229] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:06.275635 20097 raft_consensus.cc:2258] T ee19bfc2362642ae94ff2c598d60d22b P 06ae67ffe8f64168a6eb29a204b04d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:06.275646 20095 raft_consensus.cc:2229] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:06.275698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 19659
I20260927 02:48:06.276707 20095 raft_consensus.cc:2258] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:06.277921 20095 ts_tablet_manager.cc:1905] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:06.280046 20095 ts_tablet_manager.cc:1918] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:06.280153 20095 log.cc:1196] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/ee19bfc2362642ae94ff2c598d60d22b
I20260927 02:48:06.280460 20095 ts_tablet_manager.cc:1939] T ee19bfc2362642ae94ff2c598d60d22b P 953250779d3e4f539b48643fcbe290c9: Deleting consensus metadata
I20260927 02:48:06.281499 19581 catalog_manager.cc:4841] TS 953250779d3e4f539b48643fcbe290c9 (127.0.98.2:46321): tablet ee19bfc2362642ae94ff2c598d60d22b (table test-table [id=5af15b0db50d439fbb8e750fcab74a2c]) successfully deleted
W20260927 02:48:06.283532 19582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46589 (error 108)
I20260927 02:48:06.283757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 19793
I20260927 02:48:06.283763 20096 tablet_replica.cc:330] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: stopping tablet replica
I20260927 02:48:06.284102 20096 raft_consensus.cc:2229] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:06.284487 20096 raft_consensus.cc:2258] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:06.286023 20096 ts_tablet_manager.cc:1905] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:06.288655 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.288372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46589 (local address 127.0.0.1:39186)
0927 02:48:06.288498 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.288571 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 02:48:06.288826 19582 catalog_manager.cc:4584] TS 06ae67ffe8f64168a6eb29a204b04d8a (127.0.98.1:46589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee19bfc2362642ae94ff2c598d60d22b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46589: connect: Connection refused (error 111)
I20260927 02:48:06.289273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 19927
I20260927 02:48:06.289288 20096 ts_tablet_manager.cc:1918] T ee19bfc2362642ae94ff2c598d60d22b P d3502ea247e34306b25be77c54fcd741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:06.294817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 19565
2026-09-27T02:48:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2420 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T02:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:06Z Disabled control of system clock
I20260927 02:48:06.324985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33777
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:48:06.430298 20102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:06.430542 20102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:06.430574 20102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:06.434428 20102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:06.434479 20102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:06.434494 20102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:06.434510 20102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:06.434525 20102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:06.439270 20102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:06.440275 20102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:06.441529 20102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:06.449805 20107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:06.449790 20108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:06.449785 20110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:06.450093 20102 server_base.cc:1034] running on GCE node
I20260927 02:48:06.450703 20102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:06.451735 20102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:06.452922 20102 hybrid_clock.cc:648] HybridClock initialized: now 1790477286452893 us; error 50 us; skew 500 ppm
I20260927 02:48:06.454619 20102 webserver.cc:460] Webserver started at http://127.0.98.62:40285/ using document root <none> and password file <none>
I20260927 02:48:06.455196 20102 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:06.455291 20102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:06.455521 20102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:06.457533 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "753d0f6b5e464534976fede694e5fe56"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "e40dc04d2871f4467199b4072fd68488"
server_key_iv: "22d30a586bc44f271672b2d3e8a6bf67"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.458088 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "753d0f6b5e464534976fede694e5fe56"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "e40dc04d2871f4467199b4072fd68488"
server_key_iv: "22d30a586bc44f271672b2d3e8a6bf67"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.461545 20102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:48:06.463788 20116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:06.465008 20102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:06.465147 20102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "753d0f6b5e464534976fede694e5fe56"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "e40dc04d2871f4467199b4072fd68488"
server_key_iv: "22d30a586bc44f271672b2d3e8a6bf67"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.465251 20102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:06.493345 20102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:06.494036 20102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:06.494233 20102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:06.501451 20102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33777
I20260927 02:48:06.501456 20168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33777 every 8 connection(s)
I20260927 02:48:06.502404 20102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:06.505602 20169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:06.510308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 20102
I20260927 02:48:06.510443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:06.510749 392 external_mini_cluster.cc:1428] Setting key ce27ea67025bde6c5bb39e2d05fcaea2
I20260927 02:48:06.511162 20169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56: Bootstrap starting.
I20260927 02:48:06.513553 20169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:06.514321 20169 log.cc:824] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:06.516312 20169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56: No bootstrap required, opened a new log
I20260927 02:48:06.520007 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.512231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.0.98.62:33777)
0927 02:48:06.512853 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.512873 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.512908 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:48:06.513566 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.513583 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.514028 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.514301 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:06.514696 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.515626 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.516839 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.517315 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.517435 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.518263 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:06.518268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:06.518280 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:06.518648 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:06.518700 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:06.518718 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:06.518793 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.518963 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.519245 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":149,"threads_started":1}
I20260927 02:48:06.520328 20169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753d0f6b5e464534976fede694e5fe56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260927 02:48:06.520534 20169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:06.520592 20169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753d0f6b5e464534976fede694e5fe56, State: Initialized, Role: FOLLOWER
I20260927 02:48:06.521003 20169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753d0f6b5e464534976fede694e5fe56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260927 02:48:06.521152 20169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:06.521238 20169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:06.521368 20169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:06.522156 20169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753d0f6b5e464534976fede694e5fe56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260927 02:48:06.522486 20169 leader_election.cc:302] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 753d0f6b5e464534976fede694e5fe56; no voters:
I20260927 02:48:06.522763 20169 leader_election.cc:288] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:06.522889 20174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:06.523123 20174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [term 1 LEADER]: Becoming Leader. State: Replica: 753d0f6b5e464534976fede694e5fe56, State: Running, Role: LEADER
I20260927 02:48:06.523422 20174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753d0f6b5e464534976fede694e5fe56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260927 02:48:06.523934 20169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:06.525082 20175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "753d0f6b5e464534976fede694e5fe56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753d0f6b5e464534976fede694e5fe56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } } }
I20260927 02:48:06.525209 20175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:06.525396 20176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 753d0f6b5e464534976fede694e5fe56. Latest consensus state: current_term: 1 leader_uuid: "753d0f6b5e464534976fede694e5fe56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753d0f6b5e464534976fede694e5fe56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } } }
I20260927 02:48:06.525508 20176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:06.525746 20183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:06.528906 20183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:06.535983 20183 catalog_manager.cc:1347] Generated new cluster ID: 692bcbe2332a43be98ae963e1ca4a2c8
I20260927 02:48:06.536058 20183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:06.545892 20183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:06.546765 20183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:06.552337 20183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56: Generated new TSK 0
I20260927 02:48:06.552949 20183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:06.563087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:48:06.687028 20193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:06.687314 20193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:06.687390 20193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:06.691542 20193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:06.691627 20193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:06.691715 20193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:06.696385 20193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:06.697376 20193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:06.698686 20193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:06.708184 20198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:06.708237 20199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:06.708237 20201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:06.708455 20193 server_base.cc:1034] running on GCE node
I20260927 02:48:06.709097 20193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:06.709685 20193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:06.710940 20193 hybrid_clock.cc:648] HybridClock initialized: now 1790477286710916 us; error 34 us; skew 500 ppm
I20260927 02:48:06.712762 20193 webserver.cc:460] Webserver started at http://127.0.98.1:36389/ using document root <none> and password file <none>
I20260927 02:48:06.713330 20193 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:06.713431 20193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:06.713665 20193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:06.715612 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/data/instance:
uuid: "287210feeccd40b7a6f559e2e941867f"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "a7979ab8010e7ffb3680cb566b2758b4"
server_key_iv: "4cdc49341ab8b16bcebae29ae6cc18fe"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.716183 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/wal/instance:
uuid: "287210feeccd40b7a6f559e2e941867f"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "a7979ab8010e7ffb3680cb566b2758b4"
server_key_iv: "4cdc49341ab8b16bcebae29ae6cc18fe"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.719609 20193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 02:48:06.721994 20207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:06.723186 20193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:06.723328 20193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/wal
uuid: "287210feeccd40b7a6f559e2e941867f"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "a7979ab8010e7ffb3680cb566b2758b4"
server_key_iv: "4cdc49341ab8b16bcebae29ae6cc18fe"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.723453 20193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:06.750859 20193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:06.751549 20193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:06.751750 20193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:06.752444 20193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:06.753403 20193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:06.753486 20193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:06.753556 20193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:06.753605 20193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:06.762305 20193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33245
I20260927 02:48:06.762383 20320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33245 every 8 connection(s)
I20260927 02:48:06.763214 20193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:48:06.770288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 20193
I20260927 02:48:06.770435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:06.770792 392 external_mini_cluster.cc:1428] Setting key 8dbdb0922b2455d11caae17c410d729e
I20260927 02:48:06.772433 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.765071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44843 (local address 127.0.98.62:33777)
0927 02:48:06.765204 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.765208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.766248 (+ 1040us) server_negotiation.cc:407] Connection header received
0927 02:48:06.767566 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.767569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.767614 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.767685 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:06.768583 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.769125 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.770259 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.770615 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.770738 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.771941 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:06.771945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:06.771948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:06.771962 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:06.771988 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:06.771999 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:06.772067 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.772227 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.772287 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:48:06.772789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:48:06.773316 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.765459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33777 (local address 127.0.98.1:44843)
0927 02:48:06.766107 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.766147 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:06.767384 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:06.767819 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:06.767826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:06.767967 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:06.768459 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.768472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.769250 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.769254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.770130 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.770139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.770766 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.770769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.770984 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.771650 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:06.771651 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:06.771690 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:06.771748 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:06.772079 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:06.772087 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:06.772092 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:48:06.772221 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 02:48:06.772479 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
I20260927 02:48:06.774968 20321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33777
I20260927 02:48:06.775193 20321 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:06.775753 20321 heartbeater.cc:511] Master 127.0.98.62:33777 requested a full tablet report, sending...
I20260927 02:48:06.777354 20133 ts_manager.cc:194] Registered new tserver with Master: 287210feeccd40b7a6f559e2e941867f (127.0.98.1:33245)
I20260927 02:48:06.778484 20133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44843
W20260927 02:48:06.883776 20324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:06.884047 20324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:06.884083 20324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:06.888250 20324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:06.888305 20324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:06.888393 20324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:06.893042 20324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:06.894071 20324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:06.895406 20324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:06.906147 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:06.906220 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:06.906183 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:06.906430 20324 server_base.cc:1034] running on GCE node
I20260927 02:48:06.906777 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:06.907339 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:06.908552 20324 hybrid_clock.cc:648] HybridClock initialized: now 1790477286908534 us; error 31 us; skew 500 ppm
I20260927 02:48:06.910458 20324 webserver.cc:460] Webserver started at http://127.0.98.2:38251/ using document root <none> and password file <none>
I20260927 02:48:06.911027 20324 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:06.911118 20324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:06.911351 20324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:06.913396 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac07c113edb2480f9ef5ca5c5b75a509"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "dd7886093a876a5229f50bf54ddc59df"
server_key_iv: "bfdc9f12fc249d1e512b6065c26c3acb"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.913973 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac07c113edb2480f9ef5ca5c5b75a509"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "dd7886093a876a5229f50bf54ddc59df"
server_key_iv: "bfdc9f12fc249d1e512b6065c26c3acb"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.917829 20324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 02:48:06.920343 20338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:06.921742 20324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:48:06.921897 20324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/wal
uuid: "ac07c113edb2480f9ef5ca5c5b75a509"
format_stamp: "Formatted at 2026-09-27 02:48:06 on dist-test-slave-gkrl"
server_key: "dd7886093a876a5229f50bf54ddc59df"
server_key_iv: "bfdc9f12fc249d1e512b6065c26c3acb"
server_key_version: "encryptionkey@0"
I20260927 02:48:06.922013 20324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:06.931777 20324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:06.932435 20324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:06.932626 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:06.933180 20324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:06.934051 20324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:06.934121 20324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:06.934185 20324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:06.934232 20324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:06.943005 20324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34987
I20260927 02:48:06.943076 20451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34987 every 8 connection(s)
I20260927 02:48:06.943965 20324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:06.949295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 20324
I20260927 02:48:06.949407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:06.949790 392 external_mini_cluster.cc:1428] Setting key f752ac2310ad407803df21df67f673f5
I20260927 02:48:06.952045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:48:06.955068 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.945876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59085 (local address 127.0.98.62:33777)
0927 02:48:06.946031 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:06.946035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:06.947169 (+ 1134us) server_negotiation.cc:407] Connection header received
0927 02:48:06.948755 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:06.948759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:06.948823 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:06.948947 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:06.950140 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.950836 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.952719 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.952975 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.953061 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.954524 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:06.954528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:06.954531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:06.954552 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:06.954573 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:06.954582 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:06.954665 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:06.954844 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 02:48:06.954918 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:48:06.956111 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.946318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33777 (local address 127.0.98.2:59085)
0927 02:48:06.947005 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.947048 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:06.948521 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:06.949176 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:06.949184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:06.949275 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:06.949983 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.950000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.950983 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.950987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.952556 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:06.952573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:06.953154 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:06.953161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:06.953431 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:06.954191 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:06.954193 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:06.954237 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:06.954305 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:06.954681 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:06.954691 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:06.954697 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:48:06.954844 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 02:48:06.955152 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260927 02:48:06.957327 20452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33777
I20260927 02:48:06.957597 20452 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:06.958150 20452 heartbeater.cc:511] Master 127.0.98.62:33777 requested a full tablet report, sending...
I20260927 02:48:06.959249 20133 ts_manager.cc:194] Registered new tserver with Master: ac07c113edb2480f9ef5ca5c5b75a509 (127.0.98.2:34987)
I20260927 02:48:06.960182 20133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59085
W20260927 02:48:07.063465 20455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:07.063714 20455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:07.063750 20455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:07.067723 20455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:07.067776 20455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:07.067835 20455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:07.072808 20455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:07.073791 20455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:07.075081 20455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:07.084234 20463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:07.084205 20460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:07.084206 20461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:07.084609 20455 server_base.cc:1034] running on GCE node
I20260927 02:48:07.085031 20455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:07.085624 20455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:07.086822 20455 hybrid_clock.cc:648] HybridClock initialized: now 1790477287086800 us; error 46 us; skew 500 ppm
I20260927 02:48:07.091429 20455 webserver.cc:460] Webserver started at http://127.0.98.3:36043/ using document root <none> and password file <none>
I20260927 02:48:07.092113 20455 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:07.092206 20455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:07.092438 20455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:07.094404 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/data/instance:
uuid: "8d08984c40f94f2b814f1b724e1145df"
format_stamp: "Formatted at 2026-09-27 02:48:07 on dist-test-slave-gkrl"
server_key: "1fa526a1bf7011e8b2df16249b11dc1f"
server_key_iv: "0680025ccd192cbe5e39b748354f1718"
server_key_version: "encryptionkey@0"
I20260927 02:48:07.094957 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8d08984c40f94f2b814f1b724e1145df"
format_stamp: "Formatted at 2026-09-27 02:48:07 on dist-test-slave-gkrl"
server_key: "1fa526a1bf7011e8b2df16249b11dc1f"
server_key_iv: "0680025ccd192cbe5e39b748354f1718"
server_key_version: "encryptionkey@0"
I20260927 02:48:07.098487 20455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:48:07.100826 20469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:07.102028 20455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:07.102156 20455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/wal
uuid: "8d08984c40f94f2b814f1b724e1145df"
format_stamp: "Formatted at 2026-09-27 02:48:07 on dist-test-slave-gkrl"
server_key: "1fa526a1bf7011e8b2df16249b11dc1f"
server_key_iv: "0680025ccd192cbe5e39b748354f1718"
server_key_version: "encryptionkey@0"
I20260927 02:48:07.102262 20455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:07.118340 20455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:07.118985 20455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:07.119179 20455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:07.119735 20455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:07.120687 20455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:07.120761 20455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:07.120838 20455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:07.120882 20455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:07.130249 20455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36151
I20260927 02:48:07.130275 20582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36151 every 8 connection(s)
I20260927 02:48:07.131173 20455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:07.138434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 20455
I20260927 02:48:07.138587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:07.138922 392 external_mini_cluster.cc:1428] Setting key 358f0c8b955a3bc298f53c0eb13bf635
I20260927 02:48:07.140676 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.133013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54157 (local address 127.0.98.62:33777)
0927 02:48:07.133159 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:07.133164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:07.134203 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 02:48:07.135681 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:07.135685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:07.135752 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:07.135865 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:07.136808 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.137348 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.138544 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.138852 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.138972 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.140176 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:07.140179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:07.140182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:07.140195 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:07.140215 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:07.140231 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:07.140319 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:07.140477 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 02:48:07.140538 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:48:07.141680 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.133330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33777 (local address 127.0.98.3:54157)
0927 02:48:07.134029 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.134070 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:07.135447 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:07.136071 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:07.136078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:07.136165 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:07.136642 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.136655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.137482 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.137486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.138404 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.138413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.139016 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.139021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.139185 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.139853 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:07.139855 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:07.139955 (+ 100us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:07.140022 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:07.140314 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:07.140323 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:07.140331 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 02:48:07.140477 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 02:48:07.140839 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
I20260927 02:48:07.142683 20583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33777
I20260927 02:48:07.142961 20583 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:07.143474 20583 heartbeater.cc:511] Master 127.0.98.62:33777 requested a full tablet report, sending...
I20260927 02:48:07.144475 20133 ts_manager.cc:194] Registered new tserver with Master: 8d08984c40f94f2b814f1b724e1145df (127.0.98.3:36151)
I20260927 02:48:07.145033 20133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54157
I20260927 02:48:07.152854 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:48:07.158296 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.154745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.0.98.62:33777)
0927 02:48:07.154871 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:07.154875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:07.155027 (+ 152us) server_negotiation.cc:407] Connection header received
0927 02:48:07.155127 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:07.155129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:07.155164 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:07.155235 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:07.155468 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.156017 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.156825 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.157035 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.157118 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.157944 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:07.157947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:07.157949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:07.157960 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:07.157978 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:07.157985 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:07.158049 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:07.158126 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:48:07.158198 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:48:07.161083 20133 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:48:07.164167 20133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:07.178715 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.172854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33245 (local address 127.0.0.1:34838)
0927 02:48:07.174004 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.174014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:07.174080 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:07.174731 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:07.174737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:07.174758 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:07.174965 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.174973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.176429 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.176432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.177414 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.177423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.178204 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.178208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.178315 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.178349 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:48:07.178449 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 02:48:07.178561 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1105,"mutex_wait_us":86,"thread_start_us":82,"threads_started":1}
I20260927 02:48:07.178721 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.173114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.0.98.1:33245)
0927 02:48:07.173574 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:07.173586 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:07.174099 (+ 513us) server_negotiation.cc:407] Connection header received
0927 02:48:07.174173 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:07.174177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:07.174410 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:07.174568 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:07.175104 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.176300 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.177564 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.178082 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.178206 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.178413 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:07.178473 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 02:48:07.178560 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":148,"threads_started":1}
I20260927 02:48:07.179208 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.172499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34987 (local address 127.0.0.1:36460)
0927 02:48:07.173270 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.173319 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:07.173456 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:07.174819 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:07.174824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:07.174850 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:07.175202 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.175207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.176346 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.176349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.177723 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.177733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.178809 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.178813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.178931 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.178951 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:48:07.179013 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:48:07.179076 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":245,"thread_start_us":75,"threads_started":1}
I20260927 02:48:07.179497 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.172576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36151 (local address 127.0.0.1:58990)
0927 02:48:07.173744 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.173758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:07.173842 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:07.174973 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:07.174977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:07.174995 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:07.175103 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.175109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.176677 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.176680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.177847 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.177857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.178815 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.178818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.179018 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.179041 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:48:07.179106 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:48:07.179179 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":109,"thread_start_us":91,"threads_started":1}
I20260927 02:48:07.179893 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.172618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.0.98.2:34987)
0927 02:48:07.174318 (+ 1700us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:07.174330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:07.174356 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:48:07.174435 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:07.174442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:07.174642 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:07.175015 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:07.175353 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.176209 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.177874 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.178666 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.179384 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.179562 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:07.179613 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:48:07.179715 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":101,"threads_started":1}
I20260927 02:48:07.180209 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.173292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.0.98.3:36151)
0927 02:48:07.174536 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:07.174547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:07.174567 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:48:07.174635 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:07.174640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:07.174818 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:07.175365 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:07.175496 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.176584 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.177966 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.178673 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.178906 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.179856 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:07.179938 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:48:07.180047 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":150,"threads_started":1}
I20260927 02:48:07.182186 20255 tablet_service.cc:1467] Processing CreateTablet for tablet b21900d34ec44a0491d2344c7c64fc90 (DEFAULT_TABLE table=test-table [id=d165a3fa709f4617baa1047557aeb0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:07.182379 20517 tablet_service.cc:1467] Processing CreateTablet for tablet b21900d34ec44a0491d2344c7c64fc90 (DEFAULT_TABLE table=test-table [id=d165a3fa709f4617baa1047557aeb0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:07.182488 20386 tablet_service.cc:1467] Processing CreateTablet for tablet b21900d34ec44a0491d2344c7c64fc90 (DEFAULT_TABLE table=test-table [id=d165a3fa709f4617baa1047557aeb0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:07.183322 20386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21900d34ec44a0491d2344c7c64fc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:07.183323 20255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21900d34ec44a0491d2344c7c64fc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:07.183336 20517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21900d34ec44a0491d2344c7c64fc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:07.189318 20600 tablet_bootstrap.cc:492] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Bootstrap starting.
I20260927 02:48:07.190891 20601 tablet_bootstrap.cc:492] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: Bootstrap starting.
I20260927 02:48:07.190896 20602 tablet_bootstrap.cc:492] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f: Bootstrap starting.
I20260927 02:48:07.191618 20600 tablet_bootstrap.cc:654] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:07.192531 20600 log.cc:824] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:07.192638 20601 tablet_bootstrap.cc:654] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:07.193004 20602 tablet_bootstrap.cc:654] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:07.193470 20601 log.cc:824] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:07.193809 20602 log.cc:824] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:07.194572 20600 tablet_bootstrap.cc:492] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: No bootstrap required, opened a new log
I20260927 02:48:07.194782 20600 ts_tablet_manager.cc:1397] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:48:07.195633 20601 tablet_bootstrap.cc:492] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: No bootstrap required, opened a new log
I20260927 02:48:07.195863 20601 ts_tablet_manager.cc:1397] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:48:07.196174 20602 tablet_bootstrap.cc:492] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f: No bootstrap required, opened a new log
I20260927 02:48:07.196383 20602 ts_tablet_manager.cc:1397] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260927 02:48:07.198055 20600 raft_consensus.cc:348] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.198282 20600 raft_consensus.cc:374] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:07.198333 20600 raft_consensus.cc:729] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d08984c40f94f2b814f1b724e1145df, State: Initialized, Role: FOLLOWER
I20260927 02:48:07.198786 20600 consensus_queue.cc:260] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.199486 20583 heartbeater.cc:503] Master 127.0.98.62:33777 was elected leader, sending a full tablet report...
I20260927 02:48:07.199729 20600 ts_tablet_manager.cc:1428] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:48:07.200059 20601 raft_consensus.cc:348] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.200059 20602 raft_consensus.cc:348] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.200286 20601 raft_consensus.cc:374] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:07.200286 20602 raft_consensus.cc:374] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:07.200340 20601 raft_consensus.cc:729] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac07c113edb2480f9ef5ca5c5b75a509, State: Initialized, Role: FOLLOWER
I20260927 02:48:07.200345 20602 raft_consensus.cc:729] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287210feeccd40b7a6f559e2e941867f, State: Initialized, Role: FOLLOWER
I20260927 02:48:07.200784 20602 consensus_queue.cc:260] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.200784 20601 consensus_queue.cc:260] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.201696 20601 ts_tablet_manager.cc:1428] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 02:48:07.201696 20602 ts_tablet_manager.cc:1428] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 02:48:07.202052 20321 heartbeater.cc:503] Master 127.0.98.62:33777 was elected leader, sending a full tablet report...
I20260927 02:48:07.202253 20452 heartbeater.cc:503] Master 127.0.98.62:33777 was elected leader, sending a full tablet report...
W20260927 02:48:07.264403 20322 tablet.cc:2369] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:07.352684 20607 raft_consensus.cc:482] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:07.352988 20607 raft_consensus.cc:504] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.354250 20607 leader_election.cc:288] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac07c113edb2480f9ef5ca5c5b75a509 (127.0.98.2:34987), 8d08984c40f94f2b814f1b724e1145df (127.0.98.3:36151)
I20260927 02:48:07.358390 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.354640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33497 (local address 127.0.98.3:36151)
0927 02:48:07.354913 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:07.354918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:07.354944 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:48:07.355151 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:07.355154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:07.355218 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:07.355291 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:07.355664 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.356207 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.357370 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.357763 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.357887 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.357947 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:07.358177 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 02:48:07.358267 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260927 02:48:07.358395 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.354512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36151 (local address 127.0.98.1:33497)
0927 02:48:07.354864 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.354882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:07.355017 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:07.355332 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:07.355335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:07.355355 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:07.355543 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.355551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.356319 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.356323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.357220 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.357229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.357899 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.357902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.358000 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.358022 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:48:07.358133 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 02:48:07.358203 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":2,"thread_start_us":118,"threads_started":1}
I20260927 02:48:07.358568 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.354794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47367 (local address 127.0.98.2:34987)
0927 02:48:07.354932 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:07.354939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:07.355111 (+ 172us) server_negotiation.cc:407] Connection header received
0927 02:48:07.355248 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:07.355250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:07.355324 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:07.355410 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:07.355745 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.356336 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.357358 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.357859 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.357982 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.358056 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:07.358314 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:48:07.358445 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:48:07.358896 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.354512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34987 (local address 127.0.98.1:47367)
0927 02:48:07.355003 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.355021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:07.355140 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:07.355450 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:07.355452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:07.355466 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:07.355613 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.355623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.356458 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.356460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.357229 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:07.357237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.358014 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.358017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:07.358152 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.358171 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:48:07.358684 (+ 513us) client_negotiation.cc:240] Negotiation successful
0927 02:48:07.358738 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":81,"thread_start_us":74,"threads_started":1}
I20260927 02:48:07.359089 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21900d34ec44a0491d2344c7c64fc90" candidate_uuid: "287210feeccd40b7a6f559e2e941867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d08984c40f94f2b814f1b724e1145df" is_pre_election: true
I20260927 02:48:07.359447 20537 raft_consensus.cc:2454] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287210feeccd40b7a6f559e2e941867f in term 0.
I20260927 02:48:07.359615 20406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21900d34ec44a0491d2344c7c64fc90" candidate_uuid: "287210feeccd40b7a6f559e2e941867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" is_pre_election: true
I20260927 02:48:07.359968 20406 raft_consensus.cc:2454] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287210feeccd40b7a6f559e2e941867f in term 0.
I20260927 02:48:07.360008 20208 leader_election.cc:302] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287210feeccd40b7a6f559e2e941867f, 8d08984c40f94f2b814f1b724e1145df; no voters:
I20260927 02:48:07.360299 20607 raft_consensus.cc:2781] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:07.360412 20607 raft_consensus.cc:482] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:07.360476 20607 raft_consensus.cc:3037] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:07.361514 20607 raft_consensus.cc:504] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.361879 20607 leader_election.cc:288] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [CANDIDATE]: Term 1 election: Requested vote from peers ac07c113edb2480f9ef5ca5c5b75a509 (127.0.98.2:34987), 8d08984c40f94f2b814f1b724e1145df (127.0.98.3:36151)
I20260927 02:48:07.362232 20406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21900d34ec44a0491d2344c7c64fc90" candidate_uuid: "287210feeccd40b7a6f559e2e941867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac07c113edb2480f9ef5ca5c5b75a509"
I20260927 02:48:07.362260 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21900d34ec44a0491d2344c7c64fc90" candidate_uuid: "287210feeccd40b7a6f559e2e941867f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d08984c40f94f2b814f1b724e1145df"
I20260927 02:48:07.362334 20406 raft_consensus.cc:3037] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:07.362387 20537 raft_consensus.cc:3037] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:07.363356 20406 raft_consensus.cc:2454] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287210feeccd40b7a6f559e2e941867f in term 1.
I20260927 02:48:07.363360 20537 raft_consensus.cc:2454] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287210feeccd40b7a6f559e2e941867f in term 1.
I20260927 02:48:07.363639 20208 leader_election.cc:302] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287210feeccd40b7a6f559e2e941867f, 8d08984c40f94f2b814f1b724e1145df; no voters:
I20260927 02:48:07.363914 20607 raft_consensus.cc:2781] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:07.364156 20607 raft_consensus.cc:686] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 1 LEADER]: Becoming Leader. State: Replica: 287210feeccd40b7a6f559e2e941867f, State: Running, Role: LEADER
I20260927 02:48:07.364442 20607 consensus_queue.cc:237] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } }
I20260927 02:48:07.368005 20131 catalog_manager.cc:5495] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f reported cstate change: term changed from 0 to 1, leader changed from <none> to 287210feeccd40b7a6f559e2e941867f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "287210feeccd40b7a6f559e2e941867f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287210feeccd40b7a6f559e2e941867f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33245 } health_report { overall_health: HEALTHY } } }
W20260927 02:48:07.382036 20584 tablet.cc:2369] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:07.412882 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.409377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.0.98.1:33245)
0927 02:48:07.409513 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:07.409517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:07.409533 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:48:07.409662 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:07.409664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:07.409740 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:07.409860 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:07.410096 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.410611 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.411447 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:07.411682 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:07.411794 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:07.412116 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:07.412509 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:07.412643 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:07.412691 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:48:07.412741 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:48:07.418637 20406 raft_consensus.cc:1261] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 1 FOLLOWER]: Refusing update from remote peer 287210feeccd40b7a6f559e2e941867f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:07.418644 20537 raft_consensus.cc:1261] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 1 FOLLOWER]: Refusing update from remote peer 287210feeccd40b7a6f559e2e941867f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:07.419349 20611 consensus_queue.cc:1035] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d08984c40f94f2b814f1b724e1145df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:07.419595 20607 consensus_queue.cc:1035] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac07c113edb2480f9ef5ca5c5b75a509" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:07.428738 20615 mvcc.cc:205] Tried to move back new op lower bound from 7333794969259319296 to 7333794969047060480. Current Snapshot: MvccSnapshot[applied={T|T < 7333794969259319296 or (T in {7333794969259319296})}]
I20260927 02:48:07.436769 20132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:07.436978 20132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:07.439181 20132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 753d0f6b5e464534976fede694e5fe56: Sending DeleteTablet for 3 replicas of tablet b21900d34ec44a0491d2344c7c64fc90
I20260927 02:48:07.439929 20517 tablet_service.cc:1514] Processing DeleteTablet for tablet b21900d34ec44a0491d2344c7c64fc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:07 UTC) from {username='slave'} at 127.0.0.1:58990
I20260927 02:48:07.439944 20386 tablet_service.cc:1514] Processing DeleteTablet for tablet b21900d34ec44a0491d2344c7c64fc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:07 UTC) from {username='slave'} at 127.0.0.1:36460
I20260927 02:48:07.439963 20255 tablet_service.cc:1514] Processing DeleteTablet for tablet b21900d34ec44a0491d2344c7c64fc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:07 UTC) from {username='slave'} at 127.0.0.1:34838
I20260927 02:48:07.440515 20626 tablet_replica.cc:330] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: stopping tablet replica
I20260927 02:48:07.440547 20625 tablet_replica.cc:330] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f: stopping tablet replica
I20260927 02:48:07.440819 20626 raft_consensus.cc:2229] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:07.440819 20625 raft_consensus.cc:2229] T b21900d34ec44a0491d2344c7c64fc90 P 287210feeccd40b7a6f559e2e941867f [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:07.440886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 20193
I20260927 02:48:07.441094 20626 raft_consensus.cc:2258] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:07.443081 20626 ts_tablet_manager.cc:1905] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:07.443320 20624 tablet_replica.cc:330] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: stopping tablet replica
I20260927 02:48:07.443617 20624 raft_consensus.cc:2229] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:07.443926 20624 raft_consensus.cc:2258] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:07.445264 20624 ts_tablet_manager.cc:1905] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:07.445662 20626 ts_tablet_manager.cc:1918] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:07.445771 20626 log.cc:1196] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-2/wal/wals/b21900d34ec44a0491d2344c7c64fc90
I20260927 02:48:07.446055 20626 ts_tablet_manager.cc:1939] T b21900d34ec44a0491d2344c7c64fc90 P 8d08984c40f94f2b814f1b724e1145df: Deleting consensus metadata
I20260927 02:48:07.447113 20117 catalog_manager.cc:4841] TS 8d08984c40f94f2b814f1b724e1145df (127.0.98.3:36151): tablet b21900d34ec44a0491d2344c7c64fc90 (table test-table [id=d165a3fa709f4617baa1047557aeb0f4]) successfully deleted
W20260927 02:48:07.448230 20118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33245 (error 108)
I20260927 02:48:07.448499 20624 ts_tablet_manager.cc:1918] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:07.448596 20624 log.cc:1196] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: Deleting WAL directory at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477031916752-392-0/minicluster-data/ts-1/wal/wals/b21900d34ec44a0491d2344c7c64fc90
I20260927 02:48:07.448916 20624 ts_tablet_manager.cc:1939] T b21900d34ec44a0491d2344c7c64fc90 P ac07c113edb2480f9ef5ca5c5b75a509: Deleting consensus metadata
I20260927 02:48:07.449074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 20324
I20260927 02:48:07.449328 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.449035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33245 (local address 127.0.0.1:34842)
0927 02:48:07.449140 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.449214 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:48:07.449678 20118 catalog_manager.cc:4584] TS 287210feeccd40b7a6f559e2e941867f (127.0.98.1:33245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b21900d34ec44a0491d2344c7c64fc90: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33245: connect: Connection refused (error 111)
I20260927 02:48:07.455581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 20455
I20260927 02:48:07.456517 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.455717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34987 (local address 127.0.0.1:36462)
0927 02:48:07.456356 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.456421 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522}
I20260927 02:48:07.460815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 20102
2026-09-27T02:48:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1164 ms)
[----------] 6 tests from EncryptionPolicyTest (10482 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:07 dist-test-slave-gkrl krb5kdc[20630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:07 dist-test-slave-gkrl krb5kdc[20630](info): set up 2 sockets
Sep 27 02:48:07 dist-test-slave-gkrl krb5kdc[20630](info): commencing operation
krb5kdc: starting...
W20260927 02:48:08.504065 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:08 dist-test-slave-gkrl krb5kdc[20630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:08.623072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:08.729475 20643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:08.729750 20643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:08.729864 20643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:08.733830 20643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:08.733912 20643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:08.733950 20643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:08.734009 20643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:08.734048 20643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:08.739063 20643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45233
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:08.740049 20643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:08.741012 20643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:08.749058 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:08.749069 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:08.749070 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:08.749353 20643 server_base.cc:1034] running on GCE node
I20260927 02:48:08.749861 20643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:08.750867 20643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:08.752045 20643 hybrid_clock.cc:648] HybridClock initialized: now 1790477288752023 us; error 43 us; skew 500 ppm
Sep 27 02:48:08 dist-test-slave-gkrl krb5kdc[20630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:08.754520 20643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:08.755374 20643 webserver.cc:460] Webserver started at http://127.0.0.1:41253/ using document root <none> and password file <none>
I20260927 02:48:08.755954 20643 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:08.756039 20643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:08.756265 20643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:08.758157 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "3bb097fb089d417d8f29c0a4a890f491"
format_stamp: "Formatted at 2026-09-27 02:48:08 on dist-test-slave-gkrl"
server_key: "2fa81428f22c7b9e138935ac1c5157af"
server_key_iv: "90472582580d3cfd496d065061b65d90"
server_key_version: "encryptionkey@0"
I20260927 02:48:08.758704 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bb097fb089d417d8f29c0a4a890f491"
format_stamp: "Formatted at 2026-09-27 02:48:08 on dist-test-slave-gkrl"
server_key: "2fa81428f22c7b9e138935ac1c5157af"
server_key_iv: "90472582580d3cfd496d065061b65d90"
server_key_version: "encryptionkey@0"
I20260927 02:48:08.762275 20643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:48:08.764746 20658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:08.765884 20643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:08.766029 20643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "3bb097fb089d417d8f29c0a4a890f491"
format_stamp: "Formatted at 2026-09-27 02:48:08 on dist-test-slave-gkrl"
server_key: "2fa81428f22c7b9e138935ac1c5157af"
server_key_iv: "90472582580d3cfd496d065061b65d90"
server_key_version: "encryptionkey@0"
I20260927 02:48:08.766142 20643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:08.785718 20643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:08.788445 20643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:08.788647 20643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:08.796051 20643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45233
I20260927 02:48:08.796075 20710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45233 every 8 connection(s)
I20260927 02:48:08.797003 20643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:08.798601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 20643
I20260927 02:48:08.798746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:08.799048 392 external_mini_cluster.cc:1428] Setting key 05823e02d80651b439a31f86367b7d85
I20260927 02:48:08.800129 20711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:08.805435 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491: Bootstrap starting.
Sep 27 02:48:08 dist-test-slave-gkrl krb5kdc[20630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:48:08.808037 20711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:08.808950 20711 log.cc:824] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:08.811093 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491: No bootstrap required, opened a new log
I20260927 02:48:08.813648 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:08.800467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37403 (local address 127.0.0.1:45233)
0927 02:48:08.800888 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:08.800903 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:08.800937 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:48:08.801557 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:08.801572 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:08.801884 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:08.802211 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:08.802995 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:08.804024 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:08.804864 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:08.805322 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:08.805499 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:08.808445 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:08.808451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:08.808466 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:08.808495 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:08.810487 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:08.811000 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:08.811006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:08.811011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:08.811147 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:08.811514 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:08.811518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:08.811519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:08.812027 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:08.812248 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:08.812448 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:48:08.812659 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":140,"threads_started":1}
I20260927 02:48:08.815604 20711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb097fb089d417d8f29c0a4a890f491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260927 02:48:08.815843 20711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:08.815940 20711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb097fb089d417d8f29c0a4a890f491, State: Initialized, Role: FOLLOWER
I20260927 02:48:08.816409 20711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb097fb089d417d8f29c0a4a890f491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260927 02:48:08.816572 20711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:08.816644 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:08.816656 20711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:08.816747 20711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:08.817600 20711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb097fb089d417d8f29c0a4a890f491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260927 02:48:08.817948 20711 leader_election.cc:302] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb097fb089d417d8f29c0a4a890f491; no voters:
I20260927 02:48:08.818236 20711 leader_election.cc:288] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:08.818365 20720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:08.818610 20720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [term 1 LEADER]: Becoming Leader. State: Replica: 3bb097fb089d417d8f29c0a4a890f491, State: Running, Role: LEADER
I20260927 02:48:08.818938 20720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb097fb089d417d8f29c0a4a890f491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260927 02:48:08.819550 20711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:08.820106 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb097fb089d417d8f29c0a4a890f491. Latest consensus state: current_term: 1 leader_uuid: "3bb097fb089d417d8f29c0a4a890f491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb097fb089d417d8f29c0a4a890f491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } } }
I20260927 02:48:08.820257 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:08.820678 20727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:08.822177 20721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bb097fb089d417d8f29c0a4a890f491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb097fb089d417d8f29c0a4a890f491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } } }
I20260927 02:48:08.822278 20721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:08.823958 20727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:08.828178 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:08.818915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36107 (local address 127.0.0.1:45233)
0927 02:48:08.819012 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:08.819016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:08.819114 (+ 98us) server_negotiation.cc:407] Connection header received
0927 02:48:08.819197 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:08.819200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:08.819243 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:08.819344 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:08.820260 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:08.821072 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:08.822351 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:08.822683 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:08.822761 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:08.824720 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:08.824724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:08.824726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:08.824750 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:08.826348 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:08.826843 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:08.826847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:08.826848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:08.826937 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:08.827272 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:08.827275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:08.827277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:08.827450 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:08.827604 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:08.827912 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 02:48:08.828007 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:48:08.830225 20727 catalog_manager.cc:1347] Generated new cluster ID: ca7449a4f2834311bffe8a569780ef83
I20260927 02:48:08.830286 20727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:08.840174 20727 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:08.841056 20727 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:08.849663 20727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bb097fb089d417d8f29c0a4a890f491: Generated new TSK 0
I20260927 02:48:08.850260 20727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:08.857580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 20643
W20260927 02:48:08.862803 20628 connection.cc:535] client connection to 127.0.0.1:45233 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45233): Connection reset by peer (error 104)
2026-09-27T02:48:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:08 dist-test-slave-gkrl krb5kdc[20742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:08 dist-test-slave-gkrl krb5kdc[20742](info): set up 2 sockets
Sep 27 02:48:08 dist-test-slave-gkrl krb5kdc[20742](info): commencing operation
krb5kdc: starting...
W20260927 02:48:09.879988 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:09 dist-test-slave-gkrl krb5kdc[20742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:09.999970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:10.117313 20755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:10.117555 20755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:10.117595 20755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:10.121636 20755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:10.121698 20755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:10.121718 20755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:10.121735 20755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:10.121750 20755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:10.127048 20755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:10.128113 20755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:10.129091 20755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:10.137535 20763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:10.137578 20760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:10.137578 20761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:10.137758 20755 server_base.cc:1034] running on GCE node
I20260927 02:48:10.138324 20755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:10.139353 20755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:10.140535 20755 hybrid_clock.cc:648] HybridClock initialized: now 1790477290140499 us; error 55 us; skew 500 ppm
Sep 27 02:48:10 dist-test-slave-gkrl krb5kdc[20742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:10.143272 20755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:10.144101 20755 webserver.cc:460] Webserver started at http://127.0.0.1:45795/ using document root <none> and password file <none>
I20260927 02:48:10.144606 20755 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:10.144661 20755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:10.144816 20755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:10.146701 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "a1687550cf40450992cfe5f5aa6332fe"
format_stamp: "Formatted at 2026-09-27 02:48:10 on dist-test-slave-gkrl"
server_key: "121d42246e0cd6967221c90719d94ba9"
server_key_iv: "9011af1f466baeafa9b69989e8630270"
server_key_version: "encryptionkey@0"
I20260927 02:48:10.147183 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1687550cf40450992cfe5f5aa6332fe"
format_stamp: "Formatted at 2026-09-27 02:48:10 on dist-test-slave-gkrl"
server_key: "121d42246e0cd6967221c90719d94ba9"
server_key_iv: "9011af1f466baeafa9b69989e8630270"
server_key_version: "encryptionkey@0"
I20260927 02:48:10.150717 20755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 02:48:10.153033 20770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:10.154268 20755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:10.154359 20755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "a1687550cf40450992cfe5f5aa6332fe"
format_stamp: "Formatted at 2026-09-27 02:48:10 on dist-test-slave-gkrl"
server_key: "121d42246e0cd6967221c90719d94ba9"
server_key_iv: "9011af1f466baeafa9b69989e8630270"
server_key_version: "encryptionkey@0"
I20260927 02:48:10.154439 20755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:10.169741 20755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:10.172775 20755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:10.172919 20755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:10.179741 20755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38387
I20260927 02:48:10.179772 20822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38387 every 8 connection(s)
I20260927 02:48:10.180639 20755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:10.183274 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:10.186373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 20755
I20260927 02:48:10.186512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:10.186830 392 external_mini_cluster.cc:1428] Setting key 3837680e4426fcbc580be32d33f36183
I20260927 02:48:10.189188 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe: Bootstrap starting.
I20260927 02:48:10.191802 20823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:10.192606 20823 log.cc:824] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:10.194794 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe: No bootstrap required, opened a new log
I20260927 02:48:10.197120 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:10.188202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57749 (local address 127.0.0.1:38387)
0927 02:48:10.188606 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:10.188624 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:10.188665 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:48:10.189467 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:10.189490 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:10.189899 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:10.190248 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:10.190669 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:10.192115 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:10.193110 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:10.193828 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:10.194037 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:10.194863 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:10.194871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:10.194886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:10.195382 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:10.195454 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:10.195485 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:10.195643 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:10.195792 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:48:10.195952 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
I20260927 02:48:10.198454 20823 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1687550cf40450992cfe5f5aa6332fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260927 02:48:10.198694 20823 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:10.198797 20823 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1687550cf40450992cfe5f5aa6332fe, State: Initialized, Role: FOLLOWER
I20260927 02:48:10.199240 20823 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1687550cf40450992cfe5f5aa6332fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260927 02:48:10.199398 20823 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:10.199471 20823 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:10.199571 20823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:10.199955 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:10.200732 20823 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1687550cf40450992cfe5f5aa6332fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260927 02:48:10.201095 20823 leader_election.cc:302] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1687550cf40450992cfe5f5aa6332fe; no voters:
I20260927 02:48:10.201454 20823 leader_election.cc:288] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:10.201540 20831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:10.201818 20831 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [term 1 LEADER]: Becoming Leader. State: Replica: a1687550cf40450992cfe5f5aa6332fe, State: Running, Role: LEADER
I20260927 02:48:10.202163 20831 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1687550cf40450992cfe5f5aa6332fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260927 02:48:10.202796 20823 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:10.203372 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1687550cf40450992cfe5f5aa6332fe. Latest consensus state: current_term: 1 leader_uuid: "a1687550cf40450992cfe5f5aa6332fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1687550cf40450992cfe5f5aa6332fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } } }
I20260927 02:48:10.204049 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:10.203949 20833 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1687550cf40450992cfe5f5aa6332fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1687550cf40450992cfe5f5aa6332fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } } }
I20260927 02:48:10.204336 20833 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:10.205066 20842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:10.207149 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:10.202430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42815 (local address 127.0.0.1:38387)
0927 02:48:10.202526 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:10.202530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:10.202591 (+ 61us) server_negotiation.cc:407] Connection header received
0927 02:48:10.202786 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:10.202789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:10.202853 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:10.202934 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:10.203343 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:10.204206 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:10.205391 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:10.205732 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:10.205823 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:10.206684 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:10.206688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:10.206690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:10.206703 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:10.206729 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:10.206744 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:10.206868 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:10.206981 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 02:48:10.207016 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:48:10.208325 20842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:10.213645 20842 catalog_manager.cc:1347] Generated new cluster ID: 7a900c47cd7e4844b36075a568d47a24
I20260927 02:48:10.213712 20842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:10.250494 20842 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:10.251462 20842 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:10.270114 20842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1687550cf40450992cfe5f5aa6332fe: Generated new TSK 0
I20260927 02:48:10.270890 20842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:10.284852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 20755
W20260927 02:48:10.290417 20740 connection.cc:535] client connection to 127.0.0.1:38387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38387): Connection reset by peer (error 104)
2026-09-27T02:48:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:10 dist-test-slave-gkrl krb5kdc[20854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:10 dist-test-slave-gkrl krb5kdc[20854](info): set up 2 sockets
Sep 27 02:48:10 dist-test-slave-gkrl krb5kdc[20854](info): commencing operation
krb5kdc: starting...
W20260927 02:48:11.324072 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:11 dist-test-slave-gkrl krb5kdc[20854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:11.446450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:11.554206 20867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:11.554469 20867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:11.554510 20867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:11.558480 20867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:11.558537 20867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:11.558557 20867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:11.558573 20867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:11.558588 20867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:11.563848 20867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:11.564877 20867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:11.565908 20867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:11.574311 20873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:11.574334 20875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:11.574519 20867 server_base.cc:1034] running on GCE node
W20260927 02:48:11.574321 20872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:11.575191 20867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:11.576413 20867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:11.577597 20867 hybrid_clock.cc:648] HybridClock initialized: now 1790477291577568 us; error 42 us; skew 500 ppm
Sep 27 02:48:11 dist-test-slave-gkrl krb5kdc[20854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:11.580482 20867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:11.581303 20867 webserver.cc:460] Webserver started at http://127.0.0.1:40023/ using document root <none> and password file <none>
I20260927 02:48:11.581825 20867 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:11.581884 20867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:11.582043 20867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:11.584056 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "dcec5dc5327c4d8698b99419135db5db"
format_stamp: "Formatted at 2026-09-27 02:48:11 on dist-test-slave-gkrl"
server_key: "f78080895bf3849e0bdcd3f10cc5665f"
server_key_iv: "111478a83fb181b74ff7525930088ffe"
server_key_version: "encryptionkey@0"
I20260927 02:48:11.584559 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "dcec5dc5327c4d8698b99419135db5db"
format_stamp: "Formatted at 2026-09-27 02:48:11 on dist-test-slave-gkrl"
server_key: "f78080895bf3849e0bdcd3f10cc5665f"
server_key_iv: "111478a83fb181b74ff7525930088ffe"
server_key_version: "encryptionkey@0"
I20260927 02:48:11.588249 20867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:48:11.590583 20882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:11.591830 20867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:11.592008 20867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "dcec5dc5327c4d8698b99419135db5db"
format_stamp: "Formatted at 2026-09-27 02:48:11 on dist-test-slave-gkrl"
server_key: "f78080895bf3849e0bdcd3f10cc5665f"
server_key_iv: "111478a83fb181b74ff7525930088ffe"
server_key_version: "encryptionkey@0"
I20260927 02:48:11.592127 20867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:11.596539 20867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:11.599789 20867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:11.600039 20867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:11.606793 20867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33719
I20260927 02:48:11.606844 20934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33719 every 8 connection(s)
I20260927 02:48:11.607714 20867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:11.610508 20935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:11.612300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 20867
I20260927 02:48:11.612468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:11.612845 392 external_mini_cluster.cc:1428] Setting key ddaaaaa371d9aeb421f6f9db26ef4c75
I20260927 02:48:11.616983 20935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db: Bootstrap starting.
I20260927 02:48:11.617908 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:11.614203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56669 (local address 127.0.0.1:33719)
0927 02:48:11.614660 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:11.614677 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:11.614714 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:48:11.615327 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:11.615343 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:11.615696 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:11.616057 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:11.616816 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:11.616820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:11.616832 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:11.616842 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:11.616892 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:11.616909 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:11.617051 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:11.617117 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:48:11.617275 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":128,"threads_started":1}
I20260927 02:48:11.619204 20935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:11.620065 20935 log.cc:824] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:11.620507 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:11.621982 20935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db: No bootstrap required, opened a new log
I20260927 02:48:11.625504 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:11.622428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50259 (local address 127.0.0.1:33719)
0927 02:48:11.622542 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:11.622546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:11.624170 (+ 1624us) server_negotiation.cc:407] Connection header received
0927 02:48:11.624289 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:11.624292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:11.624319 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:11.624377 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:11.625079 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:11.625082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:11.625083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:11.625093 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:11.625114 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:11.625123 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:11.625203 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:11.625336 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:48:11.625364 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:48:11.626228 20935 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcec5dc5327c4d8698b99419135db5db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } }
I20260927 02:48:11.626513 20935 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:11.626603 20935 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcec5dc5327c4d8698b99419135db5db, State: Initialized, Role: FOLLOWER
I20260927 02:48:11.627027 20935 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcec5dc5327c4d8698b99419135db5db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } }
I20260927 02:48:11.627147 20935 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:11.627187 20935 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:11.627256 20935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:11.628178 20935 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcec5dc5327c4d8698b99419135db5db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } }
I20260927 02:48:11.628477 20935 leader_election.cc:302] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcec5dc5327c4d8698b99419135db5db; no voters:
I20260927 02:48:11.628711 20935 leader_election.cc:288] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:11.628820 20945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:11.629066 20945 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [term 1 LEADER]: Becoming Leader. State: Replica: dcec5dc5327c4d8698b99419135db5db, State: Running, Role: LEADER
I20260927 02:48:11.629446 20945 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcec5dc5327c4d8698b99419135db5db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } }
I20260927 02:48:11.629984 20935 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:11.630550 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcec5dc5327c4d8698b99419135db5db. Latest consensus state: current_term: 1 leader_uuid: "dcec5dc5327c4d8698b99419135db5db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcec5dc5327c4d8698b99419135db5db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } } }
I20260927 02:48:11.630745 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:11.630976 20947 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcec5dc5327c4d8698b99419135db5db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcec5dc5327c4d8698b99419135db5db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } } }
I20260927 02:48:11.631064 20947 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:11.631350 20954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:11.633971 20954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:11.639603 20954 catalog_manager.cc:1347] Generated new cluster ID: 784c9d84c2934f59912e9a823a7a771c
I20260927 02:48:11.639679 20954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:11.662178 20954 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:11.662519 20954 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:11.671852 20954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcec5dc5327c4d8698b99419135db5db: Generated new TSK 0
I20260927 02:48:11.672561 20954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:11.679510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 20867
2026-09-27T02:48:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:11 dist-test-slave-gkrl krb5kdc[20966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:11 dist-test-slave-gkrl krb5kdc[20966](info): set up 2 sockets
Sep 27 02:48:11 dist-test-slave-gkrl krb5kdc[20966](info): commencing operation
krb5kdc: starting...
W20260927 02:48:12.712173 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:12 dist-test-slave-gkrl krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:12.831219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:12.939575 20979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:12.939857 20979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:12.939944 20979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:12.943827 20979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:12.943930 20979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:12.943976 20979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:12.944013 20979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:12.944043 20979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:12.949165 20979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:12.950146 20979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:12.951114 20979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:12.959379 20985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:12.959453 20984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:12.959379 20987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:12.959543 20979 server_base.cc:1034] running on GCE node
I20260927 02:48:12.960253 20979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:12.961256 20979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:12.962452 20979 hybrid_clock.cc:648] HybridClock initialized: now 1790477292962440 us; error 36 us; skew 500 ppm
Sep 27 02:48:12 dist-test-slave-gkrl krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:12.969951 20979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:12.970790 20979 webserver.cc:460] Webserver started at http://127.0.0.1:35813/ using document root <none> and password file <none>
I20260927 02:48:12.971333 20979 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:12.971421 20979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:12.971654 20979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:12.973711 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "253c2399706648a3941224231d4f920d"
format_stamp: "Formatted at 2026-09-27 02:48:12 on dist-test-slave-gkrl"
server_key: "8a9d6a1c26c1e67d9e896e02e9e004a2"
server_key_iv: "206d78ceb55da3ccc7b1e895eee1467f"
server_key_version: "encryptionkey@0"
I20260927 02:48:12.974328 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "253c2399706648a3941224231d4f920d"
format_stamp: "Formatted at 2026-09-27 02:48:12 on dist-test-slave-gkrl"
server_key: "8a9d6a1c26c1e67d9e896e02e9e004a2"
server_key_iv: "206d78ceb55da3ccc7b1e895eee1467f"
server_key_version: "encryptionkey@0"
I20260927 02:48:12.977929 20979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:48:12.980403 20994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:12.981632 20979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:12.981796 20979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "253c2399706648a3941224231d4f920d"
format_stamp: "Formatted at 2026-09-27 02:48:12 on dist-test-slave-gkrl"
server_key: "8a9d6a1c26c1e67d9e896e02e9e004a2"
server_key_iv: "206d78ceb55da3ccc7b1e895eee1467f"
server_key_version: "encryptionkey@0"
I20260927 02:48:12.981909 20979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:12.991428 20979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:12.994058 20979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:12.994259 20979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:13.001749 20979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45025
I20260927 02:48:13.001756 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45025 every 8 connection(s)
I20260927 02:48:13.002657 20979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:13.005380 21047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:13.006805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 20979
I20260927 02:48:13.006942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:13.007256 392 external_mini_cluster.cc:1428] Setting key a0b740360cebcc57b4a34428c3ca2e88
I20260927 02:48:13.011683 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d: Bootstrap starting.
I20260927 02:48:13.014266 21047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:13.015220 21047 log.cc:824] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:48:13 dist-test-slave-gkrl krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:48:13.017344 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d: No bootstrap required, opened a new log
I20260927 02:48:13.020862 21047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253c2399706648a3941224231d4f920d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } }
I20260927 02:48:13.021184 21047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:13.021277 21047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253c2399706648a3941224231d4f920d, State: Initialized, Role: FOLLOWER
I20260927 02:48:13.021749 21047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253c2399706648a3941224231d4f920d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } }
I20260927 02:48:13.021898 21047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:13.021909 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.008575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52191 (local address 127.0.0.1:45025)
0927 02:48:13.009003 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.009018 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.009052 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:48:13.009664 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.009681 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.010015 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.010282 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.011186 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.012232 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.013380 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.013853 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.013972 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.017111 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.017116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.017128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.017164 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.019037 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.019491 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.019496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.019508 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.019614 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.019915 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.019919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.019920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.020313 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.020528 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.020767 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.021073 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":107,"threads_started":1}
I20260927 02:48:13.021999 21047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:13.022107 21047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:13.023146 21047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253c2399706648a3941224231d4f920d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } }
I20260927 02:48:13.023483 21047 leader_election.cc:302] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 253c2399706648a3941224231d4f920d; no voters:
I20260927 02:48:13.023790 21047 leader_election.cc:288] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:13.024118 21053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:13.024304 21053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [term 1 LEADER]: Becoming Leader. State: Replica: 253c2399706648a3941224231d4f920d, State: Running, Role: LEADER
I20260927 02:48:13.024629 21053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253c2399706648a3941224231d4f920d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } }
I20260927 02:48:13.024907 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:13.025043 21047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:13.025839 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 253c2399706648a3941224231d4f920d. Latest consensus state: current_term: 1 leader_uuid: "253c2399706648a3941224231d4f920d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253c2399706648a3941224231d4f920d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } } }
I20260927 02:48:13.026026 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:13.026263 21054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "253c2399706648a3941224231d4f920d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253c2399706648a3941224231d4f920d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } } }
I20260927 02:48:13.026382 21054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d [sys.catalog]: This master's current role is: LEADER
W20260927 02:48:13.030274 21071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:13.030381 21071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:13.030475 21063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:13.031378 21063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:13.035756 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.027103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57697 (local address 127.0.0.1:45025)
0927 02:48:13.027217 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.027223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.027492 (+ 269us) server_negotiation.cc:407] Connection header received
0927 02:48:13.027648 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.027651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.027690 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.027785 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.028924 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.029443 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.030246 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.030594 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.030698 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.032282 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.032286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.032288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.032313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.034029 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.034510 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.034513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.034515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.034601 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.034908 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.034910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.034912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.035121 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.035249 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.035498 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.035609 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:48:13.041181 21063 catalog_manager.cc:1347] Generated new cluster ID: 2a3acca93de0421684fb5d1dd7db60f4
I20260927 02:48:13.041251 21063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:13.057523 21063 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:13.058401 21063 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:13.064401 21063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 253c2399706648a3941224231d4f920d: Generated new TSK 0
I20260927 02:48:13.065075 21063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:13.071199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 20979
2026-09-27T02:48:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:13 dist-test-slave-gkrl krb5kdc[21078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:13 dist-test-slave-gkrl krb5kdc[21078](info): set up 2 sockets
Sep 27 02:48:13 dist-test-slave-gkrl krb5kdc[21078](info): commencing operation
krb5kdc: starting...
W20260927 02:48:14.120157 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:14 dist-test-slave-gkrl krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:14.236629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:14.347998 21091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:14.348228 21091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:14.348268 21091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:14.352936 21091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:14.352991 21091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:14.353010 21091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:14.353060 21091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:14.353096 21091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:14.358143 21091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:14.359069 21091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:14.360085 21091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:14.368177 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:14.368257 21091 server_base.cc:1034] running on GCE node
W20260927 02:48:14.368167 21099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:14.368216 21097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:14.369097 21091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:14.370216 21091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:14.371397 21091 hybrid_clock.cc:648] HybridClock initialized: now 1790477294371378 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM'
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1436: Failure
Failed
Bad status: Runtime error: failed to start masters: Unable to start Master at index 0: /tmp/dist-test-task8LsHFz/build/debug/bin/kudu: process exited with non-zero status 1
2026-09-27T02:48:32Z chronyd exiting
I20260927 02:48:32.417973 392 test_util.cc:175] -----------------------------------------------
I20260927 02:48:32.418072 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477031916752-392-0
[ FAILED ] AuthTokenIssuingTest.ChannelConfidentiality/4, where GetParam() = 32-byte object <02-00 00-00 00-00 00-00 A8-A9 06-1F 00-00 00-00 28-A5 06-1F 00-00 00-00 00-00 01-00 00-00 00-00> (19328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:32 dist-test-slave-gkrl krb5kdc[21104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:32 dist-test-slave-gkrl krb5kdc[21104](info): set up 2 sockets
Sep 27 02:48:32 dist-test-slave-gkrl krb5kdc[21104](info): commencing operation
krb5kdc: starting...
W20260927 02:48:33.424014 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:33 dist-test-slave-gkrl krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:33.542821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:33.649371 21117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:33.649639 21117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:33.649709 21117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:33.653733 21117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:33.653806 21117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:33.653843 21117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:33.653899 21117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:33.653937 21117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:33.659080 21117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:33.660111 21117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:33.661056 21117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:33.669826 21123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.669826 21122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.669898 21117 server_base.cc:1034] running on GCE node
W20260927 02:48:33.669919 21125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.670614 21117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:33.671662 21117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:33.672847 21117 hybrid_clock.cc:648] HybridClock initialized: now 1790477313672819 us; error 40 us; skew 500 ppm
Sep 27 02:48:33 dist-test-slave-gkrl krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:33.680155 21117 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:33.680971 21117 webserver.cc:460] Webserver started at http://127.0.0.1:43917/ using document root <none> and password file <none>
I20260927 02:48:33.681512 21117 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:33.681602 21117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:33.681814 21117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:33.683745 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "48084e6fc44f4ada869b5b0a4672b2d6"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-gkrl"
server_key: "97a5269fd3e2473d1384c56e99716e65"
server_key_iv: "ce204676ffd41d2af82b21ca298f9948"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.684319 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "48084e6fc44f4ada869b5b0a4672b2d6"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-gkrl"
server_key: "97a5269fd3e2473d1384c56e99716e65"
server_key_iv: "ce204676ffd41d2af82b21ca298f9948"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.687757 21117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:48:33.689978 21132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.691044 21117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:33.691181 21117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "48084e6fc44f4ada869b5b0a4672b2d6"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-gkrl"
server_key: "97a5269fd3e2473d1384c56e99716e65"
server_key_iv: "ce204676ffd41d2af82b21ca298f9948"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.691293 21117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:33.695513 21117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:33.698220 21117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:33.698407 21117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:33.705037 21117 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46735
I20260927 02:48:33.705060 21184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46735 every 8 connection(s)
I20260927 02:48:33.705884 21117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:33.708210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 21117
I20260927 02:48:33.708395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:33.708685 21185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:33.708694 392 external_mini_cluster.cc:1428] Setting key bd8f0cb5f9c86d1739aeef44b35b444f
I20260927 02:48:33.713805 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.710049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34317 (local address 127.0.0.1:46735)
0927 02:48:33.710456 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.710471 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.710509 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:48:33.711248 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.711267 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.711541 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.711818 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.712647 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:33.712652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:33.712663 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:33.712673 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:33.712724 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:33.712744 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:33.712855 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:33.712962 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:48:33.713132 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":140,"threads_started":1}
I20260927 02:48:33.715442 21185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6: Bootstrap starting.
I20260927 02:48:33.716674 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:33.717773 21185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:33.718469 21185 log.cc:824] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:33.720489 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.718775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51849 (local address 127.0.0.1:46735)
0927 02:48:33.718898 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.718905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.719119 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:48:33.719215 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.719218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.719261 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.719323 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.720055 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:33.720057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:33.720059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:33.720069 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:33.720092 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:33.720102 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:33.720172 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:33.720305 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:48:33.720337 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:48:33.720844 21185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6: No bootstrap required, opened a new log
I20260927 02:48:33.724705 21185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084e6fc44f4ada869b5b0a4672b2d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260927 02:48:33.724943 21185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:33.725031 21185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48084e6fc44f4ada869b5b0a4672b2d6, State: Initialized, Role: FOLLOWER
I20260927 02:48:33.725500 21185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084e6fc44f4ada869b5b0a4672b2d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260927 02:48:33.725698 21185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:33.725813 21185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:33.725924 21185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:33.726878 21185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084e6fc44f4ada869b5b0a4672b2d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260927 02:48:33.727169 21185 leader_election.cc:302] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48084e6fc44f4ada869b5b0a4672b2d6; no voters:
I20260927 02:48:33.727428 21185 leader_election.cc:288] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:33.727599 21195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:33.727850 21195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [term 1 LEADER]: Becoming Leader. State: Replica: 48084e6fc44f4ada869b5b0a4672b2d6, State: Running, Role: LEADER
I20260927 02:48:33.728235 21195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084e6fc44f4ada869b5b0a4672b2d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260927 02:48:33.728675 21185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:33.729894 21196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48084e6fc44f4ada869b5b0a4672b2d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084e6fc44f4ada869b5b0a4672b2d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } } }
I20260927 02:48:33.730021 21196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:33.730363 21197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48084e6fc44f4ada869b5b0a4672b2d6. Latest consensus state: current_term: 1 leader_uuid: "48084e6fc44f4ada869b5b0a4672b2d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084e6fc44f4ada869b5b0a4672b2d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } } }
I20260927 02:48:33.730433 21204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:33.730510 21197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:33.733695 21204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:33.738530 21204 catalog_manager.cc:1347] Generated new cluster ID: 6f9ce430fca6401f83431d507f7ba067
I20260927 02:48:33.738592 21204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:33.760357 21204 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:33.760700 21204 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:33.767938 21204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48084e6fc44f4ada869b5b0a4672b2d6: Generated new TSK 0
I20260927 02:48:33.768664 21204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:33.773242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 21117
2026-09-27T02:48:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:33 dist-test-slave-gkrl krb5kdc[21216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:33 dist-test-slave-gkrl krb5kdc[21216](info): set up 2 sockets
Sep 27 02:48:33 dist-test-slave-gkrl krb5kdc[21216](info): commencing operation
krb5kdc: starting...
W20260927 02:48:34.808045 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:34 dist-test-slave-gkrl krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:34.924933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:35.033278 21229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:35.033526 21229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:35.033566 21229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:35.037727 21229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:35.037782 21229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:35.037801 21229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:35.037817 21229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:35.037833 21229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:35.043011 21229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:35.044065 21229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:35.045028 21229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:35.053367 21234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:35.053381 21235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:35.053551 21229 server_base.cc:1034] running on GCE node
W20260927 02:48:35.053416 21237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:35.054126 21229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:35.055110 21229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:35.056291 21229 hybrid_clock.cc:648] HybridClock initialized: now 1790477315056263 us; error 39 us; skew 500 ppm
Sep 27 02:48:35 dist-test-slave-gkrl krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:35.058990 21229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:35.059945 21229 webserver.cc:460] Webserver started at http://127.0.0.1:43775/ using document root <none> and password file <none>
I20260927 02:48:35.060477 21229 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:35.060576 21229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:35.060799 21229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:35.062762 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "cd34ed67f59c4e229e4aa424814aa978"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-gkrl"
server_key: "62d5ea83d64c12140837a686ba3cb70a"
server_key_iv: "6ba1fbd67359ee20240c58ba15b0dc39"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.063309 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd34ed67f59c4e229e4aa424814aa978"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-gkrl"
server_key: "62d5ea83d64c12140837a686ba3cb70a"
server_key_iv: "6ba1fbd67359ee20240c58ba15b0dc39"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.066921 21229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:48:35.069345 21244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.070483 21229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:35.070626 21229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "cd34ed67f59c4e229e4aa424814aa978"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-gkrl"
server_key: "62d5ea83d64c12140837a686ba3cb70a"
server_key_iv: "6ba1fbd67359ee20240c58ba15b0dc39"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.070742 21229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:35.104602 21229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:35.107857 21229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:35.108078 21229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:35.114976 21229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38101
I20260927 02:48:35.115005 21296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38101 every 8 connection(s)
I20260927 02:48:35.115895 21229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:35.118541 21297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:35.120710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 21229
I20260927 02:48:35.120852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:35.121135 392 external_mini_cluster.cc:1428] Setting key 48ffc0a9fc66383e221d8cac90169d20
I20260927 02:48:35.124682 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978: Bootstrap starting.
I20260927 02:48:35.127096 21297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:35.127863 21297 log.cc:824] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:35.130028 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978: No bootstrap required, opened a new log
Sep 27 02:48:35 dist-test-slave-gkrl krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:48:35.134204 21297 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd34ed67f59c4e229e4aa424814aa978" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } }
I20260927 02:48:35.134469 21297 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:35.134529 21297 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd34ed67f59c4e229e4aa424814aa978, State: Initialized, Role: FOLLOWER
I20260927 02:48:35.134981 21297 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd34ed67f59c4e229e4aa424814aa978" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } }
I20260927 02:48:35.135177 21297 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:35.135289 21297 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:35.135389 21297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:35.136400 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.122426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:51545 (local address 127.0.0.1:38101)
0927 02:48:35.122808 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.122834 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.122868 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:48:35.123679 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.123697 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.124059 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.124337 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:35.125238 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.126173 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.127343 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.127791 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.127989 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.130800 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:35.130805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:35.130820 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:35.130848 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:35.133306 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.133788 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.133794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.133799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.133928 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.134222 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.134227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.134229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.134710 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:35.134917 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.135217 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.135438 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":126,"threads_started":1}
I20260927 02:48:35.136422 21297 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd34ed67f59c4e229e4aa424814aa978" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } }
I20260927 02:48:35.136749 21297 leader_election.cc:302] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd34ed67f59c4e229e4aa424814aa978; no voters:
I20260927 02:48:35.137055 21297 leader_election.cc:288] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:35.137158 21302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:35.137410 21302 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [term 1 LEADER]: Becoming Leader. State: Replica: cd34ed67f59c4e229e4aa424814aa978, State: Running, Role: LEADER
I20260927 02:48:35.137777 21302 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd34ed67f59c4e229e4aa424814aa978" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } }
I20260927 02:48:35.138262 21297 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:35.138912 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd34ed67f59c4e229e4aa424814aa978. Latest consensus state: current_term: 1 leader_uuid: "cd34ed67f59c4e229e4aa424814aa978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd34ed67f59c4e229e4aa424814aa978" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } } }
I20260927 02:48:35.139703 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:35.139739 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:35.139598 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd34ed67f59c4e229e4aa424814aa978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd34ed67f59c4e229e4aa424814aa978" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } } }
I20260927 02:48:35.140082 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978 [sys.catalog]: This master's current role is: LEADER
W20260927 02:48:35.143379 21317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:35.143462 21317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:35.143570 21312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:35.144433 21312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:35.149989 21312 catalog_manager.cc:1347] Generated new cluster ID: 8cfaf0cb5cbe40ecb7a58494f6796530
I20260927 02:48:35.150044 21312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:35.151180 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.141933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:48331 (local address 127.0.0.1:38101)
0927 02:48:35.142045 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.142049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.142319 (+ 270us) server_negotiation.cc:407] Connection header received
0927 02:48:35.142408 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.142411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.142456 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.142537 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:35.143296 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.144085 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.145171 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.145534 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.145643 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.147246 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:35.147250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:35.147253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:35.147277 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:35.149021 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.149554 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.149558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.149560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.149645 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.150072 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.150075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.150077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.150286 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:35.150433 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.150748 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.151029 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:48:35.158690 21312 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:35.159775 21312 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:35.172648 21312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd34ed67f59c4e229e4aa424814aa978: Generated new TSK 0
I20260927 02:48:35.173444 21312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:35.178676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 21229
W20260927 02:48:35.184094 21214 connection.cc:535] client connection to 127.0.0.1:38101 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38101): Connection reset by peer (error 104)
2026-09-27T02:48:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:35 dist-test-slave-gkrl krb5kdc[21328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:35 dist-test-slave-gkrl krb5kdc[21328](info): set up 2 sockets
Sep 27 02:48:35 dist-test-slave-gkrl krb5kdc[21328](info): commencing operation
krb5kdc: starting...
W20260927 02:48:36.212062 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:36 dist-test-slave-gkrl krb5kdc[21328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:36.330283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:36.434953 21341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:36.435189 21341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:36.435228 21341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:36.439136 21341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:36.439186 21341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:36.439205 21341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:36.439221 21341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:36.439280 21341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:36.444298 21341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:36.445271 21341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:36.446197 21341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:36.454198 21349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:36.454250 21341 server_base.cc:1034] running on GCE node
W20260927 02:48:36.454270 21346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:36.454205 21347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:36.455024 21341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:36.456130 21341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:36.457326 21341 hybrid_clock.cc:648] HybridClock initialized: now 1790477316457307 us; error 32 us; skew 500 ppm
Sep 27 02:48:36 dist-test-slave-gkrl krb5kdc[21328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:36.460206 21341 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:36.461093 21341 webserver.cc:460] Webserver started at http://127.0.0.1:43841/ using document root <none> and password file <none>
I20260927 02:48:36.461640 21341 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:36.461732 21341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:36.461952 21341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:36.463966 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "951406d980304211bfbf77195592d56e"
format_stamp: "Formatted at 2026-09-27 02:48:36 on dist-test-slave-gkrl"
server_key: "e9d5d76385f999af7d918d99e278906f"
server_key_iv: "bd8072f8f47da534d5c91eeb90cb126f"
server_key_version: "encryptionkey@0"
I20260927 02:48:36.464514 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "951406d980304211bfbf77195592d56e"
format_stamp: "Formatted at 2026-09-27 02:48:36 on dist-test-slave-gkrl"
server_key: "e9d5d76385f999af7d918d99e278906f"
server_key_iv: "bd8072f8f47da534d5c91eeb90cb126f"
server_key_version: "encryptionkey@0"
I20260927 02:48:36.468444 21341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 02:48:36.470794 21356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:36.472069 21341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:36.472198 21341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "951406d980304211bfbf77195592d56e"
format_stamp: "Formatted at 2026-09-27 02:48:36 on dist-test-slave-gkrl"
server_key: "e9d5d76385f999af7d918d99e278906f"
server_key_iv: "bd8072f8f47da534d5c91eeb90cb126f"
server_key_version: "encryptionkey@0"
I20260927 02:48:36.472306 21341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:36.482708 21341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:36.485512 21341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:36.485705 21341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:36.492520 21341 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40061
I20260927 02:48:36.492542 21408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40061 every 8 connection(s)
I20260927 02:48:36.493397 21341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:36.495695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 21341
I20260927 02:48:36.495857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:36.496126 392 external_mini_cluster.cc:1428] Setting key c3fffd49afd3b38557bba7b3c852ba45
I20260927 02:48:36.496145 21409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:36.503342 21409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e: Bootstrap starting.
I20260927 02:48:36.506116 21409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:36.506367 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:36.497509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:59955 (local address 127.0.0.1:40061)
0927 02:48:36.497928 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:36.497950 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:36.497993 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:48:36.498879 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:36.498903 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:36.499325 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:36.499702 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:36.500125 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:36.501568 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:36.502454 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:36.503125 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:36.503354 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:36.504323 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:36.504329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:36.504344 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:36.504862 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:36.504933 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:36.504964 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:36.505108 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:36.505317 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 02:48:36.505440 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260927 02:48:36.506947 21409 log.cc:824] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:36.508989 21409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e: No bootstrap required, opened a new log
I20260927 02:48:36.509141 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:36.512593 21409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951406d980304211bfbf77195592d56e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40061 } }
I20260927 02:48:36.512830 21409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:36.512921 21409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951406d980304211bfbf77195592d56e, State: Initialized, Role: FOLLOWER
I20260927 02:48:36.513329 21409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951406d980304211bfbf77195592d56e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40061 } }
I20260927 02:48:36.513482 21409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:36.513559 21409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:36.513685 21409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:36.514701 21409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951406d980304211bfbf77195592d56e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40061 } }
I20260927 02:48:36.515038 21409 leader_election.cc:302] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 951406d980304211bfbf77195592d56e; no voters:
I20260927 02:48:36.515307 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:36.511051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:52497 (local address 127.0.0.1:40061)
0927 02:48:36.511180 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:36.511186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:36.511426 (+ 240us) server_negotiation.cc:407] Connection header received
0927 02:48:36.511529 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:36.511533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:36.511579 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:36.511662 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:36.511961 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:36.512456 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:36.513490 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:36.513841 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:36.513974 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:36.514819 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:36.514825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:36.514830 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:36.514842 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:36.514867 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:36.514880 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:36.514981 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:36.515115 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:48:36.515148 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:48:36.515388 21409 leader_election.cc:288] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:36.515762 21420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:36.516028 21420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [term 1 LEADER]: Becoming Leader. State: Replica: 951406d980304211bfbf77195592d56e, State: Running, Role: LEADER
I20260927 02:48:36.516387 21420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951406d980304211bfbf77195592d56e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40061 } }
I20260927 02:48:36.516719 21409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:36.517871 21419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "951406d980304211bfbf77195592d56e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951406d980304211bfbf77195592d56e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40061 } } }
I20260927 02:48:36.518000 21419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:36.518366 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 951406d980304211bfbf77195592d56e. Latest consensus state: current_term: 1 leader_uuid: "951406d980304211bfbf77195592d56e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951406d980304211bfbf77195592d56e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40061 } } }
I20260927 02:48:36.518497 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e [sys.catalog]: This master's current role is: LEADER
W20260927 02:48:36.521347 21434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:36.521416 21434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:36.521507 21429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:36.522226 21429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:36.527813 21429 catalog_manager.cc:1347] Generated new cluster ID: 4f08a471a6d54fd2b55ec0f189c30689
I20260927 02:48:36.527908 21429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:36.554683 21429 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:36.555558 21429 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:36.560678 21429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 951406d980304211bfbf77195592d56e: Generated new TSK 0
I20260927 02:48:36.561443 21429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:36.572316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 21341
W20260927 02:48:36.577680 21326 connection.cc:535] client connection to 127.0.0.1:40061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40061): Connection reset by peer (error 104)
2026-09-27T02:48:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:36 dist-test-slave-gkrl krb5kdc[21440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:36 dist-test-slave-gkrl krb5kdc[21440](info): set up 2 sockets
Sep 27 02:48:36 dist-test-slave-gkrl krb5kdc[21440](info): commencing operation
krb5kdc: starting...
W20260927 02:48:37.600298 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:37 dist-test-slave-gkrl krb5kdc[21440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:37.719123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:37.827725 21453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:37.828087 21453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:37.828181 21453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:37.832186 21453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:37.832283 21453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:37.832321 21453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:37.832357 21453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:37.832391 21453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:37.837565 21453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:37.838577 21453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:37.839569 21453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:48:37.848210 21453 server_base.cc:1034] running on GCE node
W20260927 02:48:37.848165 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:37.848228 21459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:37.848141 21461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:37.848950 21453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:37.849983 21453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:37.851166 21453 hybrid_clock.cc:648] HybridClock initialized: now 1790477317851139 us; error 40 us; skew 500 ppm
Sep 27 02:48:37 dist-test-slave-gkrl krb5kdc[21440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:37.854064 21453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:37.854946 21453 webserver.cc:460] Webserver started at http://127.0.0.1:39957/ using document root <none> and password file <none>
I20260927 02:48:37.855504 21453 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:37.855564 21453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:37.855777 21453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:37.857839 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "6dae667b772542f3a242e3abe33fd050"
format_stamp: "Formatted at 2026-09-27 02:48:37 on dist-test-slave-gkrl"
server_key: "afdf5097157da0e469145ab7d106fe4a"
server_key_iv: "8e91b48ed1e398f2fac08f16ac4ec76f"
server_key_version: "encryptionkey@0"
I20260927 02:48:37.858403 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "6dae667b772542f3a242e3abe33fd050"
format_stamp: "Formatted at 2026-09-27 02:48:37 on dist-test-slave-gkrl"
server_key: "afdf5097157da0e469145ab7d106fe4a"
server_key_iv: "8e91b48ed1e398f2fac08f16ac4ec76f"
server_key_version: "encryptionkey@0"
I20260927 02:48:37.862089 21453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:48:37.864500 21468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:37.865743 21453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:37.865883 21453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "6dae667b772542f3a242e3abe33fd050"
format_stamp: "Formatted at 2026-09-27 02:48:37 on dist-test-slave-gkrl"
server_key: "afdf5097157da0e469145ab7d106fe4a"
server_key_iv: "8e91b48ed1e398f2fac08f16ac4ec76f"
server_key_version: "encryptionkey@0"
I20260927 02:48:37.865995 21453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:37.870496 21453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:37.873490 21453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:37.873683 21453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:37.880950 21453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34315
I20260927 02:48:37.880970 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34315 every 8 connection(s)
I20260927 02:48:37.881834 21453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:37.884727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 21453
I20260927 02:48:37.884877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:37.884882 21521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:37.885200 392 external_mini_cluster.cc:1428] Setting key 85f57abd3f578ace433e709dfb2cd460
I20260927 02:48:37.890342 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:37.886503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:37313 (local address 127.0.0.1:34315)
0927 02:48:37.886941 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:37.886955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:37.886989 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:48:37.887663 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:37.887679 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:37.888012 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:37.888292 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:37.889137 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:37.889141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:37.889152 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:37.889162 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:37.889227 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:37.889246 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:37.889350 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:37.889478 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:48:37.889645 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":158,"threads_started":1}
I20260927 02:48:37.892194 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050: Bootstrap starting.
I20260927 02:48:37.893265 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:37.894678 21521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:37.895561 21521 log.cc:824] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:37.897073 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:37.895351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:49475 (local address 127.0.0.1:34315)
0927 02:48:37.895457 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:37.895462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:37.895643 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:48:37.895803 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:37.895806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:37.895841 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:37.895955 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:37.896708 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:37.896712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:37.896715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:37.896728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:37.896754 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:37.896767 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:37.896860 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:37.896924 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:48:37.896961 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:48:37.897886 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050: No bootstrap required, opened a new log
I20260927 02:48:37.901947 21521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dae667b772542f3a242e3abe33fd050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260927 02:48:37.902184 21521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:37.902261 21521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dae667b772542f3a242e3abe33fd050, State: Initialized, Role: FOLLOWER
I20260927 02:48:37.902701 21521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dae667b772542f3a242e3abe33fd050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260927 02:48:37.902858 21521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:37.902953 21521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:37.903080 21521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:37.904080 21521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dae667b772542f3a242e3abe33fd050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260927 02:48:37.904430 21521 leader_election.cc:302] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dae667b772542f3a242e3abe33fd050; no voters:
I20260927 02:48:37.904783 21521 leader_election.cc:288] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:37.904904 21531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:37.905143 21531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [term 1 LEADER]: Becoming Leader. State: Replica: 6dae667b772542f3a242e3abe33fd050, State: Running, Role: LEADER
I20260927 02:48:37.905534 21531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dae667b772542f3a242e3abe33fd050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260927 02:48:37.906038 21521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:37.907226 21533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dae667b772542f3a242e3abe33fd050. Latest consensus state: current_term: 1 leader_uuid: "6dae667b772542f3a242e3abe33fd050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dae667b772542f3a242e3abe33fd050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } } }
I20260927 02:48:37.907389 21533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:37.907723 21532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dae667b772542f3a242e3abe33fd050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dae667b772542f3a242e3abe33fd050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } } }
I20260927 02:48:37.907867 21532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:37.907966 21540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:37.911016 21540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:37.916738 21540 catalog_manager.cc:1347] Generated new cluster ID: 2f0e171e8fa5490a90f75f4e1406a5d0
I20260927 02:48:37.916816 21540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:37.929764 21540 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:37.930090 21540 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:37.937709 21540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dae667b772542f3a242e3abe33fd050: Generated new TSK 0
I20260927 02:48:37.938457 21540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:37.941097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 21453
2026-09-27T02:48:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:37 dist-test-slave-gkrl krb5kdc[21552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:37 dist-test-slave-gkrl krb5kdc[21552](info): set up 2 sockets
Sep 27 02:48:37 dist-test-slave-gkrl krb5kdc[21552](info): commencing operation
krb5kdc: starting...
W20260927 02:48:38.992165 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:39 dist-test-slave-gkrl krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:39.114807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:39.220463 21565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:39.220755 21565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:39.220826 21565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:39.224754 21565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:39.224839 21565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:39.224879 21565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:39.224936 21565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:39.224977 21565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:39.230065 21565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:39.231045 21565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:39.232025 21565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:39.239985 21570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:39.239964 21573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:39.239956 21571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:39.240654 21565 server_base.cc:1034] running on GCE node
I20260927 02:48:39.241140 21565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:39.242129 21565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:39.243340 21565 hybrid_clock.cc:648] HybridClock initialized: now 1790477319243292 us; error 63 us; skew 500 ppm
Sep 27 02:48:39 dist-test-slave-gkrl krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:39.250779 21565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:39.251602 21565 webserver.cc:460] Webserver started at http://127.0.0.1:33453/ using document root <none> and password file <none>
I20260927 02:48:39.252218 21565 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:39.252314 21565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:39.252534 21565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:39.254462 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "cba90c9bde2747d68fd71ebe9e16c4a1"
format_stamp: "Formatted at 2026-09-27 02:48:39 on dist-test-slave-gkrl"
server_key: "3764c24e4c03642f00e9309cdce445a6"
server_key_iv: "463efde8f57380a58333b973b972bdb1"
server_key_version: "encryptionkey@0"
I20260927 02:48:39.255008 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "cba90c9bde2747d68fd71ebe9e16c4a1"
format_stamp: "Formatted at 2026-09-27 02:48:39 on dist-test-slave-gkrl"
server_key: "3764c24e4c03642f00e9309cdce445a6"
server_key_iv: "463efde8f57380a58333b973b972bdb1"
server_key_version: "encryptionkey@0"
I20260927 02:48:39.258564 21565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 02:48:39.260954 21580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:39.262177 21565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 02:48:39.262320 21565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "cba90c9bde2747d68fd71ebe9e16c4a1"
format_stamp: "Formatted at 2026-09-27 02:48:39 on dist-test-slave-gkrl"
server_key: "3764c24e4c03642f00e9309cdce445a6"
server_key_iv: "463efde8f57380a58333b973b972bdb1"
server_key_version: "encryptionkey@0"
I20260927 02:48:39.262429 21565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:39.281234 21565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:39.284461 21565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:39.284673 21565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:39.291417 21565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33739
I20260927 02:48:39.291502 21632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33739 every 8 connection(s)
I20260927 02:48:39.292456 21565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:39.295118 21633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:39.300441 21633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1: Bootstrap starting.
I20260927 02:48:39.300940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 21565
I20260927 02:48:39.301124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:39.301401 392 external_mini_cluster.cc:1428] Setting key 1d4ee86466294e052ac31ab6f6ce6f8c
I20260927 02:48:39.303150 21633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:39.303941 21633 log.cc:824] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:39.306027 21633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1: No bootstrap required, opened a new log
I20260927 02:48:39.309908 21633 raft_consensus.cc:348] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba90c9bde2747d68fd71ebe9e16c4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260927 02:48:39.310179 21633 raft_consensus.cc:374] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:39.310240 21633 raft_consensus.cc:729] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba90c9bde2747d68fd71ebe9e16c4a1, State: Initialized, Role: FOLLOWER
Sep 27 02:48:39 dist-test-slave-gkrl krb5kdc[21552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:48:39.310727 21633 consensus_queue.cc:260] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba90c9bde2747d68fd71ebe9e16c4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260927 02:48:39.310887 21633 raft_consensus.cc:388] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:39.310964 21633 raft_consensus.cc:482] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:39.311074 21633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:39.312050 21633 raft_consensus.cc:504] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba90c9bde2747d68fd71ebe9e16c4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260927 02:48:39.312382 21633 leader_election.cc:302] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cba90c9bde2747d68fd71ebe9e16c4a1; no voters:
I20260927 02:48:39.312777 21633 leader_election.cc:288] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:39.312881 21638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:39.313088 21638 raft_consensus.cc:686] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [term 1 LEADER]: Becoming Leader. State: Replica: cba90c9bde2747d68fd71ebe9e16c4a1, State: Running, Role: LEADER
I20260927 02:48:39.313422 21638 consensus_queue.cc:237] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba90c9bde2747d68fd71ebe9e16c4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260927 02:48:39.314036 21633 sys_catalog.cc:564] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:39.315124 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cba90c9bde2747d68fd71ebe9e16c4a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba90c9bde2747d68fd71ebe9e16c4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } } }
I20260927 02:48:39.315235 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:39.315639 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cba90c9bde2747d68fd71ebe9e16c4a1. Latest consensus state: current_term: 1 leader_uuid: "cba90c9bde2747d68fd71ebe9e16c4a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba90c9bde2747d68fd71ebe9e16c4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } } }
I20260927 02:48:39.315809 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:39.316169 21647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:39.317377 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:39.302794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:52287 (local address 127.0.0.1:33739)
0927 02:48:39.303185 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:39.303229 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:39.303272 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:48:39.303991 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:39.304031 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:39.304384 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:39.304691 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:39.305376 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:39.306829 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:39.307690 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:39.308409 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:39.308589 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:39.311366 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:39.311370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:39.311383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:39.311409 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:39.313571 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:39.314207 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:39.314213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:39.314220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:39.314364 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:39.314800 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:39.314805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:39.314808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:39.315380 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:39.315596 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:39.316199 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 02:48:39.316432 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":141,"threads_started":1}
I20260927 02:48:39.319525 21647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:39.320721 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:39.325619 21647 catalog_manager.cc:1347] Generated new cluster ID: 7f4e5fb5d200454b9fd6b5db681300a5
I20260927 02:48:39.325691 21647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:39.331218 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:39.322778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:51347 (local address 127.0.0.1:33739)
0927 02:48:39.322901 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:39.322905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:39.323122 (+ 217us) server_negotiation.cc:407] Connection header received
0927 02:48:39.323239 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:39.323242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:39.323289 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:39.323373 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:39.324133 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:39.325268 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:39.326216 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:39.326483 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:39.326594 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:39.327978 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:39.327981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:39.327984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:39.328007 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:39.329419 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:39.329957 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:39.329961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:39.329963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:39.330040 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:39.330356 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:39.330360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:39.330361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:39.330532 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:39.330683 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:39.330951 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 02:48:39.331042 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:48:39.333370 21647 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:39.334276 21647 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:39.341281 21647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cba90c9bde2747d68fd71ebe9e16c4a1: Generated new TSK 0
I20260927 02:48:39.341909 21647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:39.345593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 21565
W20260927 02:48:39.350863 21550 connection.cc:535] client connection to 127.0.0.1:33739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33739): Connection reset by peer (error 104)
2026-09-27T02:48:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:39 dist-test-slave-gkrl krb5kdc[21664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:39 dist-test-slave-gkrl krb5kdc[21664](info): set up 2 sockets
Sep 27 02:48:39 dist-test-slave-gkrl krb5kdc[21664](info): commencing operation
krb5kdc: starting...
W20260927 02:48:40.384131 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:40 dist-test-slave-gkrl krb5kdc[21664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:40.502768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:40.608911 21677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:40.609194 21677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:40.609269 21677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:40.613255 21677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:40.613333 21677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:40.613372 21677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:40.613425 21677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:40.613464 21677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:40.618562 21677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:40.619550 21677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:40.620565 21677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:40.629463 21682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:40.629530 21677 server_base.cc:1034] running on GCE node
W20260927 02:48:40.629453 21683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:40.629457 21685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:40.630298 21677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:40.631294 21677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:40.632445 21677 hybrid_clock.cc:648] HybridClock initialized: now 1790477320632424 us; error 35 us; skew 500 ppm
Sep 27 02:48:40 dist-test-slave-gkrl krb5kdc[21664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:40.635265 21677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:40.636237 21677 webserver.cc:460] Webserver started at http://127.0.0.1:36901/ using document root <none> and password file <none>
I20260927 02:48:40.636807 21677 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:40.636871 21677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:40.637096 21677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:40.639261 21677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "bfccad99f5574a97b0365c9007db559c"
format_stamp: "Formatted at 2026-09-27 02:48:40 on dist-test-slave-gkrl"
server_key: "1d29bf6c2253eb5cbdfe6e2c83877999"
server_key_iv: "e22038759ae0826ab8f57237ce50b7d9"
server_key_version: "encryptionkey@0"
I20260927 02:48:40.639849 21677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "bfccad99f5574a97b0365c9007db559c"
format_stamp: "Formatted at 2026-09-27 02:48:40 on dist-test-slave-gkrl"
server_key: "1d29bf6c2253eb5cbdfe6e2c83877999"
server_key_iv: "e22038759ae0826ab8f57237ce50b7d9"
server_key_version: "encryptionkey@0"
I20260927 02:48:40.643656 21677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:48:40.646328 21692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:40.647595 21677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:40.647758 21677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "bfccad99f5574a97b0365c9007db559c"
format_stamp: "Formatted at 2026-09-27 02:48:40 on dist-test-slave-gkrl"
server_key: "1d29bf6c2253eb5cbdfe6e2c83877999"
server_key_iv: "e22038759ae0826ab8f57237ce50b7d9"
server_key_version: "encryptionkey@0"
I20260927 02:48:40.647878 21677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:40.661937 21677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:40.665560 21677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:40.665788 21677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:40.673226 21677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40887
I20260927 02:48:40.673285 21744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40887 every 8 connection(s)
I20260927 02:48:40.674214 21677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:40.677037 21745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:40.678395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 21677
I20260927 02:48:40.678586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:40.678853 392 external_mini_cluster.cc:1428] Setting key 370395460879c17697d44406a9ad53b3
I20260927 02:48:40.683390 21745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c: Bootstrap starting.
I20260927 02:48:40.685820 21745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:40.686612 21745 log.cc:824] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:40.688786 21745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c: No bootstrap required, opened a new log
I20260927 02:48:40.688869 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:40.680254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:60933 (local address 127.0.0.1:40887)
0927 02:48:40.680665 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:40.680686 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:40.680730 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:48:40.681577 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:40.681599 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:40.682016 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:40.682382 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:40.682790 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:40.684204 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:40.685148 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:40.685825 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:40.685987 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:40.686930 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:40.686938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:40.686956 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:40.687380 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:40.687443 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:40.687483 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:40.687612 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:40.687829 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 02:48:40.687973 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":106,"threads_started":1}
I20260927 02:48:40.691817 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:40.692794 21745 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfccad99f5574a97b0365c9007db559c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260927 02:48:40.693095 21745 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:40.693158 21745 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfccad99f5574a97b0365c9007db559c, State: Initialized, Role: FOLLOWER
I20260927 02:48:40.693591 21745 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfccad99f5574a97b0365c9007db559c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260927 02:48:40.693771 21745 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:40.693849 21745 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:40.694031 21745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:40.695132 21745 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfccad99f5574a97b0365c9007db559c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260927 02:48:40.695466 21745 leader_election.cc:302] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfccad99f5574a97b0365c9007db559c; no voters:
I20260927 02:48:40.695843 21745 leader_election.cc:288] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:40.695971 21755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:40.696241 21755 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [term 1 LEADER]: Becoming Leader. State: Replica: bfccad99f5574a97b0365c9007db559c, State: Running, Role: LEADER
I20260927 02:48:40.696597 21755 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfccad99f5574a97b0365c9007db559c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260927 02:48:40.697167 21745 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:40.698655 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:40.693862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:49457 (local address 127.0.0.1:40887)
0927 02:48:40.693961 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:40.693965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:40.694163 (+ 198us) server_negotiation.cc:407] Connection header received
0927 02:48:40.694264 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:40.694268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:40.694314 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:40.694399 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:40.694652 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:40.695395 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:40.696631 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:40.696949 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:40.697054 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:40.697918 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:40.697924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:40.697929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:40.697947 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:40.697978 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:40.697994 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:40.698076 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:40.698457 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 02:48:40.698511 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:48:40.698607 21756 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfccad99f5574a97b0365c9007db559c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfccad99f5574a97b0365c9007db559c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } } }
I20260927 02:48:40.698721 21756 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:40.699020 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfccad99f5574a97b0365c9007db559c. Latest consensus state: current_term: 1 leader_uuid: "bfccad99f5574a97b0365c9007db559c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfccad99f5574a97b0365c9007db559c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } } }
I20260927 02:48:40.699203 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c [sys.catalog]: This master's current role is: LEADER
W20260927 02:48:40.702171 21770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:40.702255 21770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:40.702358 21768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:40.703158 21768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:40.708840 21768 catalog_manager.cc:1347] Generated new cluster ID: a442f7281a7044a6886c4b5cb8005fa8
I20260927 02:48:40.708918 21768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:40.720665 21768 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:40.721566 21768 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:40.736275 21768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfccad99f5574a97b0365c9007db559c: Generated new TSK 0
I20260927 02:48:40.736919 21768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:40.746136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 21677
W20260927 02:48:40.752302 21662 connection.cc:535] client connection to 127.0.0.1:40887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40887): Connection reset by peer (error 104)
2026-09-27T02:48:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:40 dist-test-slave-gkrl krb5kdc[21776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:40 dist-test-slave-gkrl krb5kdc[21776](info): set up 2 sockets
Sep 27 02:48:40 dist-test-slave-gkrl krb5kdc[21776](info): commencing operation
krb5kdc: starting...
W20260927 02:48:41.776144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:41 dist-test-slave-gkrl krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:48:41.891247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:41.998327 21789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:41.998610 21789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:41.998683 21789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:42.002795 21789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:42.002880 21789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:42.002911 21789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:42.002944 21789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:42.002979 21789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:42.011302 21789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:42.012393 21789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:42.013365 21789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:42.021868 21797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:42.021904 21794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:42.021904 21795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:42.022037 21789 server_base.cc:1034] running on GCE node
I20260927 02:48:42.022569 21789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:42.023602 21789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:42.024755 21789 hybrid_clock.cc:648] HybridClock initialized: now 1790477322024727 us; error 41 us; skew 500 ppm
Sep 27 02:48:42 dist-test-slave-gkrl krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:42.027441 21789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:48:42.028452 21789 webserver.cc:460] Webserver started at http://127.0.0.1:44583/ using document root <none> and password file <none>
I20260927 02:48:42.029150 21789 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:42.029245 21789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:42.029502 21789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:42.031569 21789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "2e1216203a3a496cbd1fd904073b5079"
format_stamp: "Formatted at 2026-09-27 02:48:42 on dist-test-slave-gkrl"
server_key: "30599e34a407eb335b61f844aede673a"
server_key_iv: "fe323a2dbb4fed049037ca1dd889cc7d"
server_key_version: "encryptionkey@0"
I20260927 02:48:42.032171 21789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e1216203a3a496cbd1fd904073b5079"
format_stamp: "Formatted at 2026-09-27 02:48:42 on dist-test-slave-gkrl"
server_key: "30599e34a407eb335b61f844aede673a"
server_key_iv: "fe323a2dbb4fed049037ca1dd889cc7d"
server_key_version: "encryptionkey@0"
I20260927 02:48:42.035938 21789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:48:42.038415 21804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:42.039727 21789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:42.039933 21789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "2e1216203a3a496cbd1fd904073b5079"
format_stamp: "Formatted at 2026-09-27 02:48:42 on dist-test-slave-gkrl"
server_key: "30599e34a407eb335b61f844aede673a"
server_key_iv: "fe323a2dbb4fed049037ca1dd889cc7d"
server_key_version: "encryptionkey@0"
I20260927 02:48:42.040064 21789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:42.044673 21789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:42.047860 21789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:42.048108 21789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:42.054491 21789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41155
I20260927 02:48:42.054538 21856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41155 every 8 connection(s)
I20260927 02:48:42.055363 21789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:42.056659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 21789
I20260927 02:48:42.056813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:42.057093 392 external_mini_cluster.cc:1428] Setting key 1a73b41e8e2dc119714bd26e84f44d10
I20260927 02:48:42.058501 21857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:42.062041 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:42.058435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:46249 (local address 127.0.0.1:41155)
0927 02:48:42.058863 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:42.058880 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:42.058916 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:48:42.059533 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:42.059549 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:42.059821 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:42.060205 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:42.060927 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:42.060933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:42.060952 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:42.060963 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:42.061013 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:42.061032 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:42.061179 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:42.061247 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:48:42.061394 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":147,"threads_started":1}
I20260927 02:48:42.064657 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079: Bootstrap starting.
I20260927 02:48:42.064776 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:42.066978 21857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:42.067732 21857 log.cc:824] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:42.068512 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:42.066741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:37255 (local address 127.0.0.1:41155)
0927 02:48:42.066844 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:42.066848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:42.067127 (+ 279us) server_negotiation.cc:407] Connection header received
0927 02:48:42.067262 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:42.067265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:42.067296 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:42.067386 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:42.068100 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:42.068104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:42.068106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:42.068118 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:42.068145 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:42.068158 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:42.068238 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:42.068364 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:48:42.068392 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:48:42.069880 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079: No bootstrap required, opened a new log
I20260927 02:48:42.073441 21857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1216203a3a496cbd1fd904073b5079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260927 02:48:42.073721 21857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:42.073796 21857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1216203a3a496cbd1fd904073b5079, State: Initialized, Role: FOLLOWER
I20260927 02:48:42.074251 21857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1216203a3a496cbd1fd904073b5079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260927 02:48:42.074383 21857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:42.074463 21857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:42.074572 21857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:42.075376 21857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1216203a3a496cbd1fd904073b5079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260927 02:48:42.075691 21857 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e1216203a3a496cbd1fd904073b5079; no voters:
I20260927 02:48:42.075990 21857 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:42.076107 21867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:42.076339 21867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [term 1 LEADER]: Becoming Leader. State: Replica: 2e1216203a3a496cbd1fd904073b5079, State: Running, Role: LEADER
I20260927 02:48:42.076623 21867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1216203a3a496cbd1fd904073b5079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260927 02:48:42.077108 21857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:42.078220 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e1216203a3a496cbd1fd904073b5079. Latest consensus state: current_term: 1 leader_uuid: "2e1216203a3a496cbd1fd904073b5079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1216203a3a496cbd1fd904073b5079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } } }
I20260927 02:48:42.078215 21868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e1216203a3a496cbd1fd904073b5079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1216203a3a496cbd1fd904073b5079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } } }
I20260927 02:48:42.078353 21868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:42.078388 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:42.078749 21876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:42.081393 21876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:42.086326 21876 catalog_manager.cc:1347] Generated new cluster ID: 6f0f609cd7794179b8bfe9fa1867234f
I20260927 02:48:42.086382 21876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:42.099253 21876 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:42.099519 21876 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:42.108584 21876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e1216203a3a496cbd1fd904073b5079: Generated new TSK 0
I20260927 02:48:42.109187 21876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:42.112880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 21789
2026-09-27T02:48:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:42 dist-test-slave-gkrl krb5kdc[21888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:42 dist-test-slave-gkrl krb5kdc[21888](info): set up 2 sockets
Sep 27 02:48:42 dist-test-slave-gkrl krb5kdc[21888](info): commencing operation
krb5kdc: starting...
W20260927 02:48:43.140087 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:43 dist-test-slave-gkrl krb5kdc[21888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:43.258250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34935
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:43.366557 21901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:43.366787 21901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:43.366828 21901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:43.370677 21901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:43.370728 21901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:43.370746 21901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:43.370762 21901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:43.370777 21901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:43.375739 21901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:43.376674 21901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:43.377553 21901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:43.385571 21907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:43.385607 21909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:43.385707 21901 server_base.cc:1034] running on GCE node
W20260927 02:48:43.385592 21906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:43.386284 21901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:43.387418 21901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:43.388615 21901 hybrid_clock.cc:648] HybridClock initialized: now 1790477323388577 us; error 48 us; skew 500 ppm
Sep 27 02:48:43 dist-test-slave-gkrl krb5kdc[21888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:43.391330 21901 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:43.392323 21901 webserver.cc:460] Webserver started at http://127.0.98.62:41177/ using document root <none> and password file <none>
I20260927 02:48:43.392879 21901 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:43.392970 21901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:43.393172 21901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:43.395066 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "ace50ba1c49d4f3cbe7bf796be07ac95"
format_stamp: "Formatted at 2026-09-27 02:48:43 on dist-test-slave-gkrl"
server_key: "db5da4172487d819dc25b60d9bfe0d97"
server_key_iv: "1d11db4244ebd5b02cb181f66fc156e9"
server_key_version: "encryptionkey@0"
I20260927 02:48:43.395612 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "ace50ba1c49d4f3cbe7bf796be07ac95"
format_stamp: "Formatted at 2026-09-27 02:48:43 on dist-test-slave-gkrl"
server_key: "db5da4172487d819dc25b60d9bfe0d97"
server_key_iv: "1d11db4244ebd5b02cb181f66fc156e9"
server_key_version: "encryptionkey@0"
I20260927 02:48:43.399250 21901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:48:43.401834 21916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:43.403012 21901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:43.403190 21901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "ace50ba1c49d4f3cbe7bf796be07ac95"
format_stamp: "Formatted at 2026-09-27 02:48:43 on dist-test-slave-gkrl"
server_key: "db5da4172487d819dc25b60d9bfe0d97"
server_key_iv: "1d11db4244ebd5b02cb181f66fc156e9"
server_key_version: "encryptionkey@0"
I20260927 02:48:43.403329 21901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:43.428002 21901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:43.431135 21901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:43.431331 21901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:43.438318 21901 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34935
I20260927 02:48:43.438344 21968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34935 every 8 connection(s)
I20260927 02:48:43.439185 21901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:43.441684 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:43.444018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 21901
I20260927 02:48:43.444166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:43.444437 392 external_mini_cluster.cc:1428] Setting key f1778e3d0eadf233f60f9c27b1d427bd
I20260927 02:48:43.447628 21969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95: Bootstrap starting.
I20260927 02:48:43.450169 21969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:43.450951 21969 log.cc:824] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:43.452816 21969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95: No bootstrap required, opened a new log
Sep 27 02:48:43 dist-test-slave-gkrl krb5kdc[21888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:43.456667 21969 raft_consensus.cc:348] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace50ba1c49d4f3cbe7bf796be07ac95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } }
I20260927 02:48:43.456923 21969 raft_consensus.cc:374] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:43.456995 21969 raft_consensus.cc:729] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace50ba1c49d4f3cbe7bf796be07ac95, State: Initialized, Role: FOLLOWER
I20260927 02:48:43.457427 21969 consensus_queue.cc:260] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace50ba1c49d4f3cbe7bf796be07ac95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } }
I20260927 02:48:43.457581 21969 raft_consensus.cc:388] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:43.457659 21969 raft_consensus.cc:482] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:43.457782 21969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:43.458801 21969 raft_consensus.cc:504] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace50ba1c49d4f3cbe7bf796be07ac95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } }
I20260927 02:48:43.459182 21969 leader_election.cc:302] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ace50ba1c49d4f3cbe7bf796be07ac95; no voters:
I20260927 02:48:43.459481 21969 leader_election.cc:288] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:43.459617 21974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:43.459846 21974 raft_consensus.cc:686] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [term 1 LEADER]: Becoming Leader. State: Replica: ace50ba1c49d4f3cbe7bf796be07ac95, State: Running, Role: LEADER
I20260927 02:48:43.460099 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:43.445837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58741 (local address 127.0.98.62:34935)
0927 02:48:43.446246 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:43.446265 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:43.446307 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:48:43.447084 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:43.447105 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:43.447486 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:43.447849 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:43.448593 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:43.450050 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:43.450966 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:43.451542 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:43.451704 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:43.454172 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:43.454178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:43.454193 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:43.454222 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:43.456816 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:43.457421 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:43.457426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:43.457431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:43.457565 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:43.457970 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:43.457976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:43.457981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:43.458434 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:43.458655 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:43.458872 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 02:48:43.459116 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20260927 02:48:43.460229 21974 consensus_queue.cc:237] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace50ba1c49d4f3cbe7bf796be07ac95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } }
I20260927 02:48:43.460656 21969 sys_catalog.cc:564] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:43.461985 21975 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ace50ba1c49d4f3cbe7bf796be07ac95. Latest consensus state: current_term: 1 leader_uuid: "ace50ba1c49d4f3cbe7bf796be07ac95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace50ba1c49d4f3cbe7bf796be07ac95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } } }
I20260927 02:48:43.462107 21975 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:43.461962 21976 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ace50ba1c49d4f3cbe7bf796be07ac95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace50ba1c49d4f3cbe7bf796be07ac95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } } }
I20260927 02:48:43.462323 21976 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:43.462488 21985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:43.463711 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:43.465730 21985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:43.471589 21985 catalog_manager.cc:1347] Generated new cluster ID: 4fa3fcee0a114f17a1652fdd5237974a
I20260927 02:48:43.471647 21985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:43.476374 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:43.468283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45217 (local address 127.0.98.62:34935)
0927 02:48:43.468382 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:43.468387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:43.468578 (+ 191us) server_negotiation.cc:407] Connection header received
0927 02:48:43.468701 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:43.468703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:43.468750 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:43.468839 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:43.469462 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:43.470095 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:43.470904 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:43.471156 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:43.471263 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:43.472912 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:43.472916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:43.472918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:43.472942 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:43.474538 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:43.475056 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:43.475060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:43.475062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:43.475145 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:43.475458 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:43.475462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:43.475463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:43.475673 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:43.475829 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:43.476126 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 02:48:43.476225 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:48:43.490872 21985 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:43.491748 21985 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:43.500844 21985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ace50ba1c49d4f3cbe7bf796be07ac95: Generated new TSK 0
I20260927 02:48:43.501507 21985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:43.513558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 21901
W20260927 02:48:43.518013 21886 connection.cc:535] client connection to 127.0.98.62:34935 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34935): Connection reset by peer (error 104)
2026-09-27T02:48:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:43 dist-test-slave-gkrl krb5kdc[22000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:43 dist-test-slave-gkrl krb5kdc[22000](info): set up 2 sockets
Sep 27 02:48:43 dist-test-slave-gkrl krb5kdc[22000](info): commencing operation
krb5kdc: starting...
W20260927 02:48:44.555961 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:44 dist-test-slave-gkrl krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:44.674649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34783
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:44.778637 22013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:44.778884 22013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:44.778924 22013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:44.782913 22013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:44.782965 22013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:44.782984 22013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:44.783041 22013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:44.783078 22013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:44.788151 22013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:44.789100 22013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:44.790139 22013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:44.798403 22019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:44.798532 22018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:44.798481 22021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:44.798785 22013 server_base.cc:1034] running on GCE node
I20260927 02:48:44.799300 22013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:44.800441 22013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:44.801633 22013 hybrid_clock.cc:648] HybridClock initialized: now 1790477324801602 us; error 45 us; skew 500 ppm
Sep 27 02:48:44 dist-test-slave-gkrl krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:44.807415 22013 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:44.808416 22013 webserver.cc:460] Webserver started at http://127.0.98.62:34525/ using document root <none> and password file <none>
I20260927 02:48:44.808977 22013 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:44.809084 22013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:44.809315 22013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:44.811319 22013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "c41f2312fc3642fcabcefa025e5420f1"
format_stamp: "Formatted at 2026-09-27 02:48:44 on dist-test-slave-gkrl"
server_key: "1c06f0f960e61181e9690e263d5c6c27"
server_key_iv: "63da03347c893a6f2256859e5698401d"
server_key_version: "encryptionkey@0"
I20260927 02:48:44.811868 22013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "c41f2312fc3642fcabcefa025e5420f1"
format_stamp: "Formatted at 2026-09-27 02:48:44 on dist-test-slave-gkrl"
server_key: "1c06f0f960e61181e9690e263d5c6c27"
server_key_iv: "63da03347c893a6f2256859e5698401d"
server_key_version: "encryptionkey@0"
I20260927 02:48:44.815461 22013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:48:44.817832 22028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:44.818979 22013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:48:44.819126 22013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "c41f2312fc3642fcabcefa025e5420f1"
format_stamp: "Formatted at 2026-09-27 02:48:44 on dist-test-slave-gkrl"
server_key: "1c06f0f960e61181e9690e263d5c6c27"
server_key_iv: "63da03347c893a6f2256859e5698401d"
server_key_version: "encryptionkey@0"
I20260927 02:48:44.819235 22013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:44.846793 22013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:44.849534 22013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:44.849726 22013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:44.856526 22013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34783
I20260927 02:48:44.856562 22080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34783 every 8 connection(s)
I20260927 02:48:44.857401 22013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:44.859992 22081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:44.860000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22013
I20260927 02:48:44.860144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:44.860430 392 external_mini_cluster.cc:1428] Setting key 362cdad34acc3babc343240c1776460d
I20260927 02:48:44.866714 22081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1: Bootstrap starting.
I20260927 02:48:44.869366 22081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:44.870195 22081 log.cc:824] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:44.870296 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:44.861762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43581 (local address 127.0.98.62:34783)
0927 02:48:44.862161 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:44.862180 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:44.862221 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:48:44.863154 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:44.863178 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:44.863590 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:44.863970 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:44.864351 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:44.865803 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:44.866731 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:44.867372 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:44.867521 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:44.868407 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:44.868413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:44.868427 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:44.868892 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:44.868958 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:44.868987 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:44.869113 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:44.869271 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 02:48:44.869404 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260927 02:48:44.872503 22081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1: No bootstrap required, opened a new log
I20260927 02:48:44.873412 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:44.876569 22081 raft_consensus.cc:348] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41f2312fc3642fcabcefa025e5420f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34783 } }
I20260927 02:48:44.876812 22081 raft_consensus.cc:374] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:44.876886 22081 raft_consensus.cc:729] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41f2312fc3642fcabcefa025e5420f1, State: Initialized, Role: FOLLOWER
I20260927 02:48:44.877324 22081 consensus_queue.cc:260] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41f2312fc3642fcabcefa025e5420f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34783 } }
I20260927 02:48:44.877483 22081 raft_consensus.cc:388] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:44.877542 22081 raft_consensus.cc:482] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:44.877631 22081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:44.878741 22081 raft_consensus.cc:504] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41f2312fc3642fcabcefa025e5420f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34783 } }
I20260927 02:48:44.879106 22081 leader_election.cc:302] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c41f2312fc3642fcabcefa025e5420f1; no voters:
I20260927 02:48:44.879366 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:44.875334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51407 (local address 127.0.98.62:34783)
0927 02:48:44.875454 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:44.875459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:44.875636 (+ 177us) server_negotiation.cc:407] Connection header received
0927 02:48:44.875733 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:44.875735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:44.875773 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:44.875836 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:44.876151 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:44.876643 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:44.877778 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:44.878015 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:44.878097 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:44.878918 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:44.878921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:44.878924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:44.878942 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:44.878965 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:44.878975 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:44.879043 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:44.879195 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 02:48:44.879244 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:48:44.879475 22081 leader_election.cc:288] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:44.879552 22091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:44.879741 22091 raft_consensus.cc:686] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [term 1 LEADER]: Becoming Leader. State: Replica: c41f2312fc3642fcabcefa025e5420f1, State: Running, Role: LEADER
I20260927 02:48:44.880102 22091 consensus_queue.cc:237] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41f2312fc3642fcabcefa025e5420f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34783 } }
I20260927 02:48:44.880709 22081 sys_catalog.cc:564] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:44.881219 22091 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c41f2312fc3642fcabcefa025e5420f1. Latest consensus state: current_term: 1 leader_uuid: "c41f2312fc3642fcabcefa025e5420f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41f2312fc3642fcabcefa025e5420f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34783 } } }
I20260927 02:48:44.881389 22091 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:44.881793 22098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:44.881903 22092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c41f2312fc3642fcabcefa025e5420f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41f2312fc3642fcabcefa025e5420f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34783 } } }
I20260927 02:48:44.882040 22092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:44.884934 22098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:44.890228 22098 catalog_manager.cc:1347] Generated new cluster ID: d268e7653bb9485497bdbff86ae75feb
I20260927 02:48:44.890332 22098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:44.899475 22098 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:44.900643 22098 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:44.909896 22098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c41f2312fc3642fcabcefa025e5420f1: Generated new TSK 0
I20260927 02:48:44.910665 22098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:44.920202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22013
W20260927 02:48:44.926270 21998 connection.cc:535] client connection to 127.0.98.62:34783 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34783): Connection reset by peer (error 104)
2026-09-27T02:48:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:44 dist-test-slave-gkrl krb5kdc[22112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:44 dist-test-slave-gkrl krb5kdc[22112](info): set up 2 sockets
Sep 27 02:48:44 dist-test-slave-gkrl krb5kdc[22112](info): commencing operation
krb5kdc: starting...
W20260927 02:48:45.948256 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:46 dist-test-slave-gkrl krb5kdc[22112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:46.073268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:46.179760 22125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:46.180022 22125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:46.180061 22125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:46.183902 22125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:46.183950 22125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:46.183969 22125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:46.183986 22125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:46.184001 22125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:46.189038 22125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:46.189942 22125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:46.190856 22125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:46.199122 22133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:46.199147 22131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:46.199146 22130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:46.199250 22125 server_base.cc:1034] running on GCE node
I20260927 02:48:46.200080 22125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:46.201104 22125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:46.202265 22125 hybrid_clock.cc:648] HybridClock initialized: now 1790477326202245 us; error 31 us; skew 500 ppm
Sep 27 02:48:46 dist-test-slave-gkrl krb5kdc[22112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:46.204932 22125 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:46.205760 22125 webserver.cc:460] Webserver started at http://127.0.98.62:36187/ using document root <none> and password file <none>
I20260927 02:48:46.206297 22125 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:46.206386 22125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:46.206616 22125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:46.208742 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "4d2c837980984f7a9735433ff4089a7e"
format_stamp: "Formatted at 2026-09-27 02:48:46 on dist-test-slave-gkrl"
server_key: "79a853c1e0086e739cf0f72eacaa96ac"
server_key_iv: "0baab26abab4e0af29aa7834039b9123"
server_key_version: "encryptionkey@0"
I20260927 02:48:46.209278 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d2c837980984f7a9735433ff4089a7e"
format_stamp: "Formatted at 2026-09-27 02:48:46 on dist-test-slave-gkrl"
server_key: "79a853c1e0086e739cf0f72eacaa96ac"
server_key_iv: "0baab26abab4e0af29aa7834039b9123"
server_key_version: "encryptionkey@0"
I20260927 02:48:46.212853 22125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:48:46.215119 22140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:46.216298 22125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:46.216432 22125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "4d2c837980984f7a9735433ff4089a7e"
format_stamp: "Formatted at 2026-09-27 02:48:46 on dist-test-slave-gkrl"
server_key: "79a853c1e0086e739cf0f72eacaa96ac"
server_key_iv: "0baab26abab4e0af29aa7834039b9123"
server_key_version: "encryptionkey@0"
I20260927 02:48:46.216537 22125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:46.220917 22125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:46.223515 22125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:46.223737 22125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:46.230511 22125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36377
I20260927 02:48:46.230540 22192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36377 every 8 connection(s)
I20260927 02:48:46.231341 22125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:46.233932 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:46.238811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22125
I20260927 02:48:46.238950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:46.239199 392 external_mini_cluster.cc:1428] Setting key 538279ebca224459b6dadd048680bc86
I20260927 02:48:46.239216 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e: Bootstrap starting.
I20260927 02:48:46.241904 22193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:46.242743 22193 log.cc:824] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:46.244335 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:46.240500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57539 (local address 127.0.98.62:36377)
0927 02:48:46.240894 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:46.240911 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:46.240950 (+ 39us) server_negotiation.cc:407] Connection header received
0927 02:48:46.241755 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:46.241782 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:46.242140 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:46.242489 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:46.243149 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:46.243154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:46.243168 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:46.243181 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:46.243240 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:46.243259 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:46.243383 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:46.243488 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:48:46.243639 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
I20260927 02:48:46.244771 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e: No bootstrap required, opened a new log
I20260927 02:48:46.246955 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:46.248329 22193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c837980984f7a9735433ff4089a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36377 } }
I20260927 02:48:46.248546 22193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:46.248626 22193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2c837980984f7a9735433ff4089a7e, State: Initialized, Role: FOLLOWER
I20260927 02:48:46.249083 22193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c837980984f7a9735433ff4089a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36377 } }
I20260927 02:48:46.249244 22193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:46.249356 22193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:46.249470 22193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:46.250558 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:46.248936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34661 (local address 127.0.98.62:36377)
0927 02:48:46.249046 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:46.249050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:46.249269 (+ 219us) server_negotiation.cc:407] Connection header received
0927 02:48:46.249403 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:46.249405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:46.249457 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:46.249541 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:46.250202 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:46.250205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:46.250207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:46.250216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:46.250238 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:46.250247 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:46.250314 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:46.250421 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:48:46.250447 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:48:46.250598 22193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c837980984f7a9735433ff4089a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36377 } }
I20260927 02:48:46.250945 22193 leader_election.cc:302] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d2c837980984f7a9735433ff4089a7e; no voters:
I20260927 02:48:46.251266 22193 leader_election.cc:288] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:46.251353 22203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:46.251569 22203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [term 1 LEADER]: Becoming Leader. State: Replica: 4d2c837980984f7a9735433ff4089a7e, State: Running, Role: LEADER
I20260927 02:48:46.251940 22203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c837980984f7a9735433ff4089a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36377 } }
I20260927 02:48:46.252473 22193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:46.253492 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d2c837980984f7a9735433ff4089a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c837980984f7a9735433ff4089a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36377 } } }
I20260927 02:48:46.253612 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:46.253774 22205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d2c837980984f7a9735433ff4089a7e. Latest consensus state: current_term: 1 leader_uuid: "4d2c837980984f7a9735433ff4089a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c837980984f7a9735433ff4089a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36377 } } }
I20260927 02:48:46.254074 22205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:46.254707 22212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:46.257613 22212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:46.262780 22212 catalog_manager.cc:1347] Generated new cluster ID: 6cb8ba51ebaa4d86ab136195871f61c9
I20260927 02:48:46.262850 22212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:46.290773 22212 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:46.291095 22212 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:46.298761 22212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d2c837980984f7a9735433ff4089a7e: Generated new TSK 0
I20260927 02:48:46.299414 22212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:46.306537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22125
2026-09-27T02:48:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:46 dist-test-slave-gkrl krb5kdc[22224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:46 dist-test-slave-gkrl krb5kdc[22224](info): set up 2 sockets
Sep 27 02:48:46 dist-test-slave-gkrl krb5kdc[22224](info): commencing operation
krb5kdc: starting...
W20260927 02:48:47.344194 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:47 dist-test-slave-gkrl krb5kdc[22224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:47.463222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:47.571556 22237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:47.571790 22237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:47.571830 22237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:47.575750 22237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:47.575809 22237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:47.575829 22237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:47.575845 22237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:47.575860 22237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:47.581002 22237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:47.581904 22237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:47.582831 22237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:47.590900 22242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:47.590910 22243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:47.590982 22237 server_base.cc:1034] running on GCE node
W20260927 02:48:47.590919 22245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:47.591720 22237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:47.592762 22237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:47.593916 22237 hybrid_clock.cc:648] HybridClock initialized: now 1790477327593897 us; error 31 us; skew 500 ppm
Sep 27 02:48:47 dist-test-slave-gkrl krb5kdc[22224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:47.596704 22237 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:47.597543 22237 webserver.cc:460] Webserver started at http://127.0.98.62:39913/ using document root <none> and password file <none>
I20260927 02:48:47.598068 22237 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:47.598129 22237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:47.598340 22237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:47.600376 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "30dc3e2fded6444db1612924849fbbf6"
format_stamp: "Formatted at 2026-09-27 02:48:47 on dist-test-slave-gkrl"
server_key: "0c9a7ff0aef34bef9d08f3120ac8a7a1"
server_key_iv: "948fe237d61726cf73e84dbfc8f1ed05"
server_key_version: "encryptionkey@0"
I20260927 02:48:47.600934 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "30dc3e2fded6444db1612924849fbbf6"
format_stamp: "Formatted at 2026-09-27 02:48:47 on dist-test-slave-gkrl"
server_key: "0c9a7ff0aef34bef9d08f3120ac8a7a1"
server_key_iv: "948fe237d61726cf73e84dbfc8f1ed05"
server_key_version: "encryptionkey@0"
I20260927 02:48:47.604523 22237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:48:47.606820 22252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:47.607997 22237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 02:48:47.608114 22237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "30dc3e2fded6444db1612924849fbbf6"
format_stamp: "Formatted at 2026-09-27 02:48:47 on dist-test-slave-gkrl"
server_key: "0c9a7ff0aef34bef9d08f3120ac8a7a1"
server_key_iv: "948fe237d61726cf73e84dbfc8f1ed05"
server_key_version: "encryptionkey@0"
I20260927 02:48:47.608218 22237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:47.625355 22237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:47.628130 22237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:47.628319 22237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:47.635043 22237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44243
I20260927 02:48:47.635064 22304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44243 every 8 connection(s)
I20260927 02:48:47.636014 22237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:47.638608 22305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:47.638773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22237
I20260927 02:48:47.638919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:47.639192 392 external_mini_cluster.cc:1428] Setting key 26b055da84d961c5b722d93820e28d8b
I20260927 02:48:47.644634 22305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6: Bootstrap starting.
I20260927 02:48:47.646827 22305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:47.647639 22305 log.cc:824] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:48:47 dist-test-slave-gkrl krb5kdc[22224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:47.649742 22305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6: No bootstrap required, opened a new log
I20260927 02:48:47.653147 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:47.640475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41295 (local address 127.0.98.62:44243)
0927 02:48:47.640890 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:47.640904 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:47.640937 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:48:47.641728 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:47.641744 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:47.642077 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:47.642407 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:47.643185 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:47.644218 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:47.645107 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:47.645598 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:47.645763 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:47.648492 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:47.648497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:47.648513 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:47.648543 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:47.650386 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:47.650852 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:47.650862 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:47.650866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:47.650972 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:47.651275 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:47.651279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:47.651281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:47.651659 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:47.651856 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:47.652140 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 02:48:47.652430 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":142,"threads_started":1}
I20260927 02:48:47.653290 22305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3e2fded6444db1612924849fbbf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260927 02:48:47.653620 22305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:47.653692 22305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30dc3e2fded6444db1612924849fbbf6, State: Initialized, Role: FOLLOWER
I20260927 02:48:47.654134 22305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3e2fded6444db1612924849fbbf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260927 02:48:47.654322 22305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:47.654443 22305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:47.654574 22305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:47.655920 22305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3e2fded6444db1612924849fbbf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260927 02:48:47.656306 22305 leader_election.cc:302] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30dc3e2fded6444db1612924849fbbf6; no voters:
I20260927 02:48:47.656423 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:47.656621 22305 leader_election.cc:288] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:47.656726 22310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:47.656914 22310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [term 1 LEADER]: Becoming Leader. State: Replica: 30dc3e2fded6444db1612924849fbbf6, State: Running, Role: LEADER
I20260927 02:48:47.657310 22310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3e2fded6444db1612924849fbbf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260927 02:48:47.657891 22305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:47.658521 22310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30dc3e2fded6444db1612924849fbbf6. Latest consensus state: current_term: 1 leader_uuid: "30dc3e2fded6444db1612924849fbbf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3e2fded6444db1612924849fbbf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } } }
I20260927 02:48:47.658699 22310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:47.659020 22311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30dc3e2fded6444db1612924849fbbf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3e2fded6444db1612924849fbbf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } } }
I20260927 02:48:47.659139 22311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:47.659317 22324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:47.662513 22324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:47.668874 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:47.658585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57799 (local address 127.0.98.62:44243)
0927 02:48:47.658704 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:47.658709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:47.660283 (+ 1574us) server_negotiation.cc:407] Connection header received
0927 02:48:47.660422 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:47.660428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:47.660477 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:47.660561 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:47.661376 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:47.662122 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:47.663236 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:47.663555 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:47.663669 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:47.665425 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:47.665429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:47.665432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:47.665458 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:47.667047 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:47.667587 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:47.667589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:47.667590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:47.667672 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:47.667981 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:47.667986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:47.667988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:47.668198 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:47.668320 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:47.668600 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:48:47.668735 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:48:47.669574 22324 catalog_manager.cc:1347] Generated new cluster ID: c3bc37e25b124c8e9d33042fd610df94
I20260927 02:48:47.669625 22324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:47.680445 22324 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:47.681391 22324 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:47.685595 22324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30dc3e2fded6444db1612924849fbbf6: Generated new TSK 0
I20260927 02:48:47.686370 22324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:47.698581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22237
2026-09-27T02:48:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:47 dist-test-slave-gkrl krb5kdc[22336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:47 dist-test-slave-gkrl krb5kdc[22336](info): set up 2 sockets
Sep 27 02:48:47 dist-test-slave-gkrl krb5kdc[22336](info): commencing operation
krb5kdc: starting...
W20260927 02:48:48.744097 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:48 dist-test-slave-gkrl krb5kdc[22336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:48.862422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33671
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:48.970798 22349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:48.971058 22349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:48.971099 22349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:48.975028 22349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:48.975085 22349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:48.975104 22349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:48.975121 22349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:48.975135 22349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:48.980361 22349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:48.981315 22349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:48.982257 22349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:48.990525 22357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:48.990512 22354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:48.990599 22355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:48.990926 22349 server_base.cc:1034] running on GCE node
I20260927 02:48:48.991448 22349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:48.992588 22349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:48.993800 22349 hybrid_clock.cc:648] HybridClock initialized: now 1790477328993795 us; error 48 us; skew 500 ppm
Sep 27 02:48:48 dist-test-slave-gkrl krb5kdc[22336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:48.996692 22349 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:48.997648 22349 webserver.cc:460] Webserver started at http://127.0.98.62:38631/ using document root <none> and password file <none>
I20260927 02:48:48.998172 22349 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:48.998231 22349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:48.998392 22349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:49.000499 22349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "5d30fd6b752f45e2afce4846243741f9"
format_stamp: "Formatted at 2026-09-27 02:48:48 on dist-test-slave-gkrl"
server_key: "f717cb52356d026ee289cde0c66fee98"
server_key_iv: "fc96e982593de3a83597bb5ca2c4333d"
server_key_version: "encryptionkey@0"
I20260927 02:48:49.001015 22349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d30fd6b752f45e2afce4846243741f9"
format_stamp: "Formatted at 2026-09-27 02:48:48 on dist-test-slave-gkrl"
server_key: "f717cb52356d026ee289cde0c66fee98"
server_key_iv: "fc96e982593de3a83597bb5ca2c4333d"
server_key_version: "encryptionkey@0"
I20260927 02:48:49.004755 22349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 02:48:49.007110 22364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:49.008440 22349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:49.008565 22349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "5d30fd6b752f45e2afce4846243741f9"
format_stamp: "Formatted at 2026-09-27 02:48:48 on dist-test-slave-gkrl"
server_key: "f717cb52356d026ee289cde0c66fee98"
server_key_iv: "fc96e982593de3a83597bb5ca2c4333d"
server_key_version: "encryptionkey@0"
I20260927 02:48:49.008678 22349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:49.033579 22349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:49.037333 22349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:49.037557 22349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:49.045547 22349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33671
I20260927 02:48:49.045634 22416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33671 every 8 connection(s)
I20260927 02:48:49.046597 22349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:49.048288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22349
I20260927 02:48:49.048435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:49.048733 392 external_mini_cluster.cc:1428] Setting key dd3de1781f472844c8a3e7caec45c4b2
I20260927 02:48:49.049708 22417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:49.056761 22417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9: Bootstrap starting.
I20260927 02:48:49.059355 22417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:49.059389 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:49.050200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51381 (local address 127.0.98.62:33671)
0927 02:48:49.050622 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:49.050641 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:49.050681 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:48:49.051499 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:49.051534 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:49.051978 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:49.052349 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:49.052762 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:49.054230 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:49.055498 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:49.056221 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:49.056378 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:49.057391 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:49.057397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:49.057415 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:49.057974 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:49.058060 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:49.058094 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:49.058221 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:49.058348 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 02:48:49.058572 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260927 02:48:49.060168 22417 log.cc:824] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:49.062117 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:49.062202 22417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9: No bootstrap required, opened a new log
I20260927 02:48:49.066318 22417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d30fd6b752f45e2afce4846243741f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33671 } }
I20260927 02:48:49.066594 22417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:49.066685 22417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d30fd6b752f45e2afce4846243741f9, State: Initialized, Role: FOLLOWER
I20260927 02:48:49.067116 22417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d30fd6b752f45e2afce4846243741f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33671 } }
I20260927 02:48:49.067279 22417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:49.067358 22417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:49.067471 22417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:49.068455 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:49.064090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39667 (local address 127.0.98.62:33671)
0927 02:48:49.064219 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:49.064223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:49.064353 (+ 130us) server_negotiation.cc:407] Connection header received
0927 02:48:49.064478 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:49.064480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:49.064515 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:49.064576 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:49.064908 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:49.065410 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:49.066620 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:49.066970 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:49.067061 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:49.068006 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:49.068009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:49.068012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:49.068023 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:49.068050 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:49.068066 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:49.068144 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:49.068255 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 02:48:49.068304 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:48:49.068444 22417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d30fd6b752f45e2afce4846243741f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33671 } }
I20260927 02:48:49.068809 22417 leader_election.cc:302] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d30fd6b752f45e2afce4846243741f9; no voters:
I20260927 02:48:49.069164 22417 leader_election.cc:288] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:49.069231 22427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:49.069463 22427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [term 1 LEADER]: Becoming Leader. State: Replica: 5d30fd6b752f45e2afce4846243741f9, State: Running, Role: LEADER
I20260927 02:48:49.069779 22427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d30fd6b752f45e2afce4846243741f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33671 } }
I20260927 02:48:49.070451 22417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:49.071544 22429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d30fd6b752f45e2afce4846243741f9. Latest consensus state: current_term: 1 leader_uuid: "5d30fd6b752f45e2afce4846243741f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d30fd6b752f45e2afce4846243741f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33671 } } }
I20260927 02:48:49.071547 22428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d30fd6b752f45e2afce4846243741f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d30fd6b752f45e2afce4846243741f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33671 } } }
I20260927 02:48:49.071772 22428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:49.071764 22429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:49.072206 22436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:49.075523 22436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:49.081055 22436 catalog_manager.cc:1347] Generated new cluster ID: f2de0afcbe1d40e781b6c37f1f17e066
I20260927 02:48:49.081118 22436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:49.123580 22436 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:49.124904 22436 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:49.132503 22436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d30fd6b752f45e2afce4846243741f9: Generated new TSK 0
I20260927 02:48:49.133296 22436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:49.144681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22349
2026-09-27T02:48:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:49 dist-test-slave-gkrl krb5kdc[22448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:49 dist-test-slave-gkrl krb5kdc[22448](info): set up 2 sockets
Sep 27 02:48:49 dist-test-slave-gkrl krb5kdc[22448](info): commencing operation
krb5kdc: starting...
W20260927 02:48:50.180203 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:50 dist-test-slave-gkrl krb5kdc[22448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:50.299117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:50.403452 22461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:50.403692 22461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:50.403731 22461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:50.407570 22461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:50.407624 22461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:50.407642 22461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:50.407658 22461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:50.407673 22461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:50.412606 22461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:50.413465 22461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:50.414356 22461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:50.422569 22467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:50.422585 22461 server_base.cc:1034] running on GCE node
W20260927 02:48:50.422540 22466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:50.422533 22469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:50.423182 22461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:50.424226 22461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:50.425379 22461 hybrid_clock.cc:648] HybridClock initialized: now 1790477330425353 us; error 42 us; skew 500 ppm
Sep 27 02:48:50 dist-test-slave-gkrl krb5kdc[22448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:50.432190 22461 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:50.433039 22461 webserver.cc:460] Webserver started at http://127.0.98.62:39851/ using document root <none> and password file <none>
I20260927 02:48:50.433513 22461 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:50.433574 22461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:50.433725 22461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:50.435550 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "a142ca062b954d3abce5a84478b4e90e"
format_stamp: "Formatted at 2026-09-27 02:48:50 on dist-test-slave-gkrl"
server_key: "e3cf22145ce4f8d503f2625e3fd36ab4"
server_key_iv: "9bd621333841f9da83f749904064fd48"
server_key_version: "encryptionkey@0"
I20260927 02:48:50.436101 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "a142ca062b954d3abce5a84478b4e90e"
format_stamp: "Formatted at 2026-09-27 02:48:50 on dist-test-slave-gkrl"
server_key: "e3cf22145ce4f8d503f2625e3fd36ab4"
server_key_iv: "9bd621333841f9da83f749904064fd48"
server_key_version: "encryptionkey@0"
I20260927 02:48:50.439400 22461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 02:48:50.441684 22476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:50.442793 22461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:50.442924 22461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "a142ca062b954d3abce5a84478b4e90e"
format_stamp: "Formatted at 2026-09-27 02:48:50 on dist-test-slave-gkrl"
server_key: "e3cf22145ce4f8d503f2625e3fd36ab4"
server_key_iv: "9bd621333841f9da83f749904064fd48"
server_key_version: "encryptionkey@0"
I20260927 02:48:50.443027 22461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:50.447289 22461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:50.449955 22461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:50.450137 22461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:50.456699 22461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44669
I20260927 02:48:50.456725 22528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44669 every 8 connection(s)
I20260927 02:48:50.457504 22461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:50.460069 22529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:50.464200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22461
I20260927 02:48:50.464362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:50.464636 392 external_mini_cluster.cc:1428] Setting key c9e5083e76ced2ff29d8487415f9409e
I20260927 02:48:50.465394 22529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e: Bootstrap starting.
I20260927 02:48:50.467691 22529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:50.468590 22529 log.cc:824] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:50.469534 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:50.465936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56923 (local address 127.0.98.62:44669)
0927 02:48:50.466379 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:50.466397 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:50.466431 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:48:50.467041 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:50.467069 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:50.467331 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:50.467667 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:50.468417 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:50.468422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:50.468436 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:50.468453 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:50.468500 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:50.468517 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:50.468633 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:50.468758 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 02:48:50.468901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260927 02:48:50.471374 22529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e: No bootstrap required, opened a new log
I20260927 02:48:50.472213 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:50.475343 22529 raft_consensus.cc:348] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a142ca062b954d3abce5a84478b4e90e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260927 02:48:50.475759 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:50.474151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46261 (local address 127.0.98.62:44669)
0927 02:48:50.474271 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:50.474278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:50.474472 (+ 194us) server_negotiation.cc:407] Connection header received
0927 02:48:50.474597 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:50.474599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:50.474623 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:50.474684 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:50.475336 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:50.475338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:50.475340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:50.475350 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:50.475372 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:50.475381 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:50.475447 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:50.475557 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:48:50.475588 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:48:50.475633 22529 raft_consensus.cc:374] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:50.475905 22529 raft_consensus.cc:729] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a142ca062b954d3abce5a84478b4e90e, State: Initialized, Role: FOLLOWER
I20260927 02:48:50.476330 22529 consensus_queue.cc:260] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a142ca062b954d3abce5a84478b4e90e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260927 02:48:50.476449 22529 raft_consensus.cc:388] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:50.476652 22529 raft_consensus.cc:482] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:50.476753 22529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:50.477572 22529 raft_consensus.cc:504] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a142ca062b954d3abce5a84478b4e90e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260927 02:48:50.477900 22529 leader_election.cc:302] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a142ca062b954d3abce5a84478b4e90e; no voters:
I20260927 02:48:50.478209 22529 leader_election.cc:288] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:50.478277 22539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:50.478528 22539 raft_consensus.cc:686] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [term 1 LEADER]: Becoming Leader. State: Replica: a142ca062b954d3abce5a84478b4e90e, State: Running, Role: LEADER
I20260927 02:48:50.478927 22539 consensus_queue.cc:237] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a142ca062b954d3abce5a84478b4e90e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260927 02:48:50.479372 22529 sys_catalog.cc:564] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:50.480401 22541 sys_catalog.cc:455] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a142ca062b954d3abce5a84478b4e90e. Latest consensus state: current_term: 1 leader_uuid: "a142ca062b954d3abce5a84478b4e90e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a142ca062b954d3abce5a84478b4e90e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } } }
I20260927 02:48:50.480535 22541 sys_catalog.cc:458] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:50.480948 22548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:50.481269 22540 sys_catalog.cc:455] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a142ca062b954d3abce5a84478b4e90e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a142ca062b954d3abce5a84478b4e90e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } } }
I20260927 02:48:50.481389 22540 sys_catalog.cc:458] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:50.483686 22548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:50.488376 22548 catalog_manager.cc:1347] Generated new cluster ID: 99970b7dac154d15b890857e4180b34d
I20260927 02:48:50.488433 22548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:50.507274 22548 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:50.507582 22548 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:50.516642 22548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a142ca062b954d3abce5a84478b4e90e: Generated new TSK 0
I20260927 02:48:50.517266 22548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:50.528168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22461
2026-09-27T02:48:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1383 ms)
[----------] 18 tests from AuthTokenIssuingTest (43065 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:50 dist-test-slave-gkrl krb5kdc[22560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:50 dist-test-slave-gkrl krb5kdc[22560](info): set up 2 sockets
Sep 27 02:48:50 dist-test-slave-gkrl krb5kdc[22560](info): commencing operation
krb5kdc: starting...
W20260927 02:48:51.560130 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:51 dist-test-slave-gkrl krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:51.678918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36901
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36901,127.0.98.61:43649,127.0.98.60:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:51.783560 22573 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:48:51.783865 22573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:51.783955 22573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:51.784029 22573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:51.787942 22573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:51.788028 22573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:51.788069 22573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:51.788120 22573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:51.788157 22573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:51.793138 22573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36901,127.0.98.61:43649,127.0.98.60:37469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:51.794077 22573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:51.795004 22573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:51.803046 22579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:51.803090 22578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:51.803073 22581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:51.803221 22573 server_base.cc:1034] running on GCE node
I20260927 02:48:51.804073 22573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:51.805109 22573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:51.806288 22573 hybrid_clock.cc:648] HybridClock initialized: now 1790477331806250 us; error 55 us; skew 500 ppm
Sep 27 02:48:51 dist-test-slave-gkrl krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:51.808957 22573 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:51.809785 22573 webserver.cc:460] Webserver started at http://127.0.98.62:33439/ using document root <none> and password file <none>
I20260927 02:48:51.810315 22573 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:51.810374 22573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:51.810613 22573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:51.812651 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "f1199f443c734667a2d845d8671906fe"
format_stamp: "Formatted at 2026-09-27 02:48:51 on dist-test-slave-gkrl"
server_key: "b3eac981f60a4e82206beb382d1d92ff"
server_key_iv: "d2ada0d99df5061e97be25d77b6eaf92"
server_key_version: "encryptionkey@0"
I20260927 02:48:51.813184 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1199f443c734667a2d845d8671906fe"
format_stamp: "Formatted at 2026-09-27 02:48:51 on dist-test-slave-gkrl"
server_key: "b3eac981f60a4e82206beb382d1d92ff"
server_key_iv: "d2ada0d99df5061e97be25d77b6eaf92"
server_key_version: "encryptionkey@0"
I20260927 02:48:51.816694 22573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:48:51.818920 22588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:51.820091 22573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:51.820233 22573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "f1199f443c734667a2d845d8671906fe"
format_stamp: "Formatted at 2026-09-27 02:48:51 on dist-test-slave-gkrl"
server_key: "b3eac981f60a4e82206beb382d1d92ff"
server_key_iv: "d2ada0d99df5061e97be25d77b6eaf92"
server_key_version: "encryptionkey@0"
I20260927 02:48:51.820339 22573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:51.848362 22573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:51.851046 22573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:51.851250 22573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:51.857848 22573 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36901
I20260927 02:48:51.858121 22640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36901 every 8 connection(s)
I20260927 02:48:51.858784 22573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:51.861392 22641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:51.864828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22573
I20260927 02:48:51.864959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:51.865248 392 external_mini_cluster.cc:1428] Setting key 99c0e3abdc2064a80a41c1120737b8d5
I20260927 02:48:51.865074 22641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 02:48:51 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:51.881801 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:51.867619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.0.98.62:36901)
0927 02:48:51.868052 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:51.868071 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:51.868325 (+ 254us) server_negotiation.cc:407] Connection header received
0927 02:48:51.869347 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:51.869351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:51.869546 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:51.869752 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:51.870574 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:51.871436 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:51.872564 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:51.872955 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:51.873138 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:51.876857 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:51.876868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:51.876884 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:51.876915 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:51.878674 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:51.879262 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:51.879267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:51.879268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:51.879368 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:51.879789 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:51.879792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:51.879793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:51.880213 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:51.880320 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:51.880810 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 02:48:51.881046 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260927 02:48:51.881807 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:51.867620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36901 (local address 127.0.0.1:59972)
0927 02:48:51.868188 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:51.868222 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:51.869059 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:51.869746 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:51.869751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:51.870197 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:51.870426 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:51.870438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:51.871560 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:51.871563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:51.872313 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:51.872322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:51.873122 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:51.873125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:51.873277 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:51.873904 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:51.873921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:51.876700 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:51.878795 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:51.878807 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:51.878821 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:51.879139 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:51.879469 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:51.879475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:51.879480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:51.879647 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:51.880339 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:51.880348 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:51.880529 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 02:48:51.880737 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 02:48:51.881149 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":94,"spinlock_wait_cycles":24448,"thread_start_us":152,"threads_started":1}
I20260927 02:48:51.884290 22641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:51.885020 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:51.884795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33600)
0927 02:48:51.884902 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:51.884966 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:48:51.885263 22590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
I20260927 02:48:51.886124 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:51.885916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33616)
0927 02:48:51.886033 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:51.886075 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 02:48:51.886298 22641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:48:51.890834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43649
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36901,127.0.98.61:43649,127.0.98.60:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:48:51.935567 22641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } attempt: 1
I20260927 02:48:51.936666 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:51.936187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33618)
0927 02:48:51.936452 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:51.936572 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 02:48:51.937507 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:51.937262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33626)
0927 02:48:51.937375 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:51.937430 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:48:51.937682 22641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
W20260927 02:48:52.000730 22648 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:48:52.001000 22648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:52.001027 22648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:52.001135 22648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:52.005401 22648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:52.005543 22648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:52.005611 22648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:52.005656 22648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:52.005697 22648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:52.005888 22641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } attempt: 2
I20260927 02:48:52.006903 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.006494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33640)
0927 02:48:52.006728 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.006797 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 02:48:52.007815 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.007499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33644)
0927 02:48:52.007644 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.007730 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 02:48:52.008203 22641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43649: connect: Connection refused (error 111)
I20260927 02:48:52.013638 22648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36901,127.0.98.61:43649,127.0.98.60:37469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:52.014761 22648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:52.015820 22648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:52.024212 22656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:52.024251 22654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:52.024228 22653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:52.024509 22648 server_base.cc:1034] running on GCE node
I20260927 02:48:52.025068 22648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:52.026199 22648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:52.027395 22648 hybrid_clock.cc:648] HybridClock initialized: now 1790477332027368 us; error 48 us; skew 500 ppm
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:52.030230 22648 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:48:52.031082 22648 webserver.cc:460] Webserver started at http://127.0.98.61:37023/ using document root <none> and password file <none>
I20260927 02:48:52.031620 22648 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:52.031708 22648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:52.031999 22648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:52.033989 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/data/instance:
uuid: "3d0fbfb98d1b4c349fe3e650f15c803c"
format_stamp: "Formatted at 2026-09-27 02:48:52 on dist-test-slave-gkrl"
server_key: "f10dcc9687e9a344332c1321bea511e2"
server_key_iv: "2b139848777c6a00bb06f305b9f76295"
server_key_version: "encryptionkey@0"
I20260927 02:48:52.034551 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/wal/instance:
uuid: "3d0fbfb98d1b4c349fe3e650f15c803c"
format_stamp: "Formatted at 2026-09-27 02:48:52 on dist-test-slave-gkrl"
server_key: "f10dcc9687e9a344332c1321bea511e2"
server_key_iv: "2b139848777c6a00bb06f305b9f76295"
server_key_version: "encryptionkey@0"
I20260927 02:48:52.038221 22648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:48:52.040603 22663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:52.041776 22648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:52.041916 22648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/wal
uuid: "3d0fbfb98d1b4c349fe3e650f15c803c"
format_stamp: "Formatted at 2026-09-27 02:48:52 on dist-test-slave-gkrl"
server_key: "f10dcc9687e9a344332c1321bea511e2"
server_key_iv: "2b139848777c6a00bb06f305b9f76295"
server_key_version: "encryptionkey@0"
I20260927 02:48:52.042022 22648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:52.079039 22648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:52.082230 22648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:52.082429 22648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:52.089354 22648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43649
I20260927 02:48:52.089636 22715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43649 every 8 connection(s)
I20260927 02:48:52.090337 22648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:48:52.093176 22716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:52.096745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22648
I20260927 02:48:52.096872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-1/wal/instance
I20260927 02:48:52.097146 392 external_mini_cluster.cc:1428] Setting key db27e6bcadc3896e1906390b948f3bc8
I20260927 02:48:52.096865 22716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:52.099412 22641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:52.114778 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.099862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33656)
0927 02:48:52.100048 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.100062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:52.100161 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:52.101850 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:52.101854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:52.102094 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:52.102261 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.102274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.103580 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.103585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.104585 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.104599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.105540 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.105543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.105677 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.106226 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:52.106239 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:52.109951 (+ 3712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:52.112737 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.112740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.112742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.112934 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.113339 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.113343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.113347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.113456 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.114174 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:52.114177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:52.114276 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 02:48:52.114400 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 02:48:52.114554 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":22}
I20260927 02:48:52.115653 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.098769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.0.98.62:36901)
0927 02:48:52.098898 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.098902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.100124 (+ 1222us) server_negotiation.cc:407] Connection header received
0927 02:48:52.101655 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.101658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.101706 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.101792 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.103472 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.104086 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.105333 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.105995 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.106107 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.110831 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.110836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.110843 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.110873 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.113703 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.114325 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.114329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.114330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.114409 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.114711 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.114714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.114715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.114903 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.114977 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.115386 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.115492 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":45}
I20260927 02:48:52.115991 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.100306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.0.98.61:43649)
0927 02:48:52.100616 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.100630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.100667 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:48:52.101401 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.101424 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.101659 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.101839 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.102450 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.103454 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.104727 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.105422 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.105592 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.110153 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.110163 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.110178 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.110212 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.112586 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.113054 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.113060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.113065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.113221 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.113592 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.113598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.113600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.114044 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.114204 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.114611 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.114927 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":215,"thread_start_us":150,"threads_started":1}
I20260927 02:48:52.115993 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.099249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36901 (local address 127.0.0.1:59986)
0927 02:48:52.099952 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.100013 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:52.101395 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:52.102508 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:52.102517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:52.103041 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:52.103294 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.103312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.104216 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.104221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.105198 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.105208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.106292 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.106298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.106440 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.106997 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:52.107013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:52.110638 (+ 3625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:52.113815 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.113826 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.113835 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.114163 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.114512 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.114518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.114523 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.114608 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.115063 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:52.115069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:52.115178 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:48:52.115321 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 02:48:52.115494 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":61,"thread_start_us":152,"threads_started":1}
I20260927 02:48:52.117347 22716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:52.118120 22641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:52.118788 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.118544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:59950)
0927 02:48:52.118670 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.118723 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 02:48:52.119464 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.119213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:59966)
0927 02:48:52.119376 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.119418 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 02:48:52.119625 22641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37469: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:48:52.125200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37469
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36901,127.0.98.61:43649,127.0.98.60:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:48:52.126235 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.117974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.0.98.61:43649)
0927 02:48:52.118287 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.118291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.118300 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:52.118331 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.118333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.118363 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.118453 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.118894 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.119578 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.120458 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.120694 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.120775 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.122934 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.122937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.122938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.122958 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.124508 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.124966 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.124968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.124969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.125038 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.125331 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.125336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.125339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.125553 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.125651 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.125956 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.126109 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":257}
I20260927 02:48:52.126355 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.117857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33672)
0927 02:48:52.118060 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.118073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:52.118145 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:52.118481 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:52.118483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:52.118674 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:52.118770 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.118777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.119749 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.119755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.120348 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.120357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.120802 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.120805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.120877 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.121423 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:52.121436 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:52.122808 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:52.124600 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.124605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.124607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.124857 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.125127 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.125131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.125133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.125227 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.125661 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:52.125667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:52.125782 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 02:48:52.125897 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 02:48:52.126056 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260927 02:48:52.127092 22716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:52.127775 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.127541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:59982)
0927 02:48:52.127649 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.127730 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 02:48:52.128005 22666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
I20260927 02:48:52.128784 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.128593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:59990)
0927 02:48:52.128694 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.128742 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:48:52.128945 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37469: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
I20260927 02:48:52.150802 22641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } attempt: 1
I20260927 02:48:52.151634 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.151294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60000)
0927 02:48:52.151465 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.151536 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 02:48:52.152483 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.152249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60016)
0927 02:48:52.152361 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.152421 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:48:52.152642 22641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37469: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
I20260927 02:48:52.178239 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } attempt: 1
I20260927 02:48:52.179044 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.178759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60028)
0927 02:48:52.178903 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.178970 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:48:52.179778 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.179572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60032)
0927 02:48:52.179677 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.179729 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:48:52.179968 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37469: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
I20260927 02:48:52.227834 22641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } attempt: 2
I20260927 02:48:52.228741 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.228399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60044)
0927 02:48:52.228567 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.228650 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 02:48:52.229566 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.229320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60056)
0927 02:48:52.229439 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.229496 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 02:48:52.229835 22641 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37469: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
W20260927 02:48:52.235236 22722 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:48:52.235507 22722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:52.235579 22722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:52.235649 22722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:52.239627 22722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:52.239678 22722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:52.239698 22722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:52.239715 22722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:52.239770 22722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:52.245083 22722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36901,127.0.98.61:43649,127.0.98.60:37469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:52.246165 22722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:52.247138 22722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:48:52.248142 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } attempt: 2
I20260927 02:48:52.248903 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.248580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60058)
0927 02:48:52.248753 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.248817 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 02:48:52.249677 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.249418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60062)
0927 02:48:52.249541 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.249597 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 02:48:52.249818 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37469: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37469: connect: Connection refused (error 111)
W20260927 02:48:52.255453 22731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:52.255437 22729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:52.255463 22728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:52.255726 22722 server_base.cc:1034] running on GCE node
I20260927 02:48:52.256381 22722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:52.257453 22722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:52.258628 22722 hybrid_clock.cc:648] HybridClock initialized: now 1790477332258596 us; error 46 us; skew 500 ppm
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:52.261437 22722 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:48:52.262225 22722 webserver.cc:460] Webserver started at http://127.0.98.60:43231/ using document root <none> and password file <none>
I20260927 02:48:52.262746 22722 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:52.262832 22722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:52.263046 22722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:52.265018 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/data/instance:
uuid: "088045bd960048e39f54f6fdd0acd6ec"
format_stamp: "Formatted at 2026-09-27 02:48:52 on dist-test-slave-gkrl"
server_key: "1e8297d531327f9b7fb7729904dc52e7"
server_key_iv: "0d2b17283e893a1eb9c7bc76c0c3d4df"
server_key_version: "encryptionkey@0"
I20260927 02:48:52.265576 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/wal/instance:
uuid: "088045bd960048e39f54f6fdd0acd6ec"
format_stamp: "Formatted at 2026-09-27 02:48:52 on dist-test-slave-gkrl"
server_key: "1e8297d531327f9b7fb7729904dc52e7"
server_key_iv: "0d2b17283e893a1eb9c7bc76c0c3d4df"
server_key_version: "encryptionkey@0"
I20260927 02:48:52.269044 22722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:48:52.271404 22738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:52.272626 22722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:48:52.272759 22722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/wal
uuid: "088045bd960048e39f54f6fdd0acd6ec"
format_stamp: "Formatted at 2026-09-27 02:48:52 on dist-test-slave-gkrl"
server_key: "1e8297d531327f9b7fb7729904dc52e7"
server_key_iv: "0d2b17283e893a1eb9c7bc76c0c3d4df"
server_key_version: "encryptionkey@0"
I20260927 02:48:52.272866 22722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:52.294211 22722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:52.297436 22722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:52.297667 22722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:52.304955 22722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37469
I20260927 02:48:52.305270 22790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37469 every 8 connection(s)
I20260927 02:48:52.306033 22722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:48:52.308931 22791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:52.312155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22722
I20260927 02:48:52.312278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477031916752-392-0/minicluster-data/master-2/wal/instance
I20260927 02:48:52.312561 392 external_mini_cluster.cc:1428] Setting key 34a8bdff1b1855b1559d58b32ef678cd
I20260927 02:48:52.312868 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:48:52.325503 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.314885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.0.98.62:36901)
0927 02:48:52.315050 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.315054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.315321 (+ 267us) server_negotiation.cc:407] Connection header received
0927 02:48:52.315712 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.315723 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.315790 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.315909 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.316740 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.317289 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.318349 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.318628 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.318754 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.321495 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.321498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.321500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.321524 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.323337 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.323864 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.323869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.323897 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.323992 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.324384 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.324389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.324391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.324651 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.324768 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.325185 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.325307 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 02:48:52.326274 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.313898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.0.98.60:37469)
0927 02:48:52.314356 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.314399 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.314442 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:48:52.315303 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.315324 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.315577 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.315778 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.316616 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.317471 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.318310 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.318771 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.318932 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.321788 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.321793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.321802 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.321822 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.323532 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.324029 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.324037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.324043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.324197 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.324476 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.324479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.324480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.324824 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.325022 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.325242 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.325427 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"spinlock_wait_cycles":12544,"thread_start_us":129,"threads_started":1}
I20260927 02:48:52.326753 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.314744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36901 (local address 127.0.0.1:60000)
0927 02:48:52.315184 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.315224 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:52.315519 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:52.315972 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:52.315978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:52.316336 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:52.316592 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.316606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.317409 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.317416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.318216 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.318225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.318788 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.318792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.318931 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.319498 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:52.319514 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:52.321257 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:52.323481 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.323490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.323499 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.323770 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.324147 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.324150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.324153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.324276 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.324771 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:52.324781 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:52.324925 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 02:48:52.325111 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 02:48:52.326630 (+ 1519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":1222,"spinlock_wait_cycles":4864,"thread_start_us":109,"threads_started":1}
I20260927 02:48:52.327837 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:52.329008 22641 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } attempt: 3
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:48:52.337797 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.328419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43649 (local address 127.0.0.1:33674)
0927 02:48:52.328527 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.328544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:52.328622 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:52.328879 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:52.328882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:52.329079 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:52.329258 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.329269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.330216 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.330218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.331072 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.331083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.331613 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.331617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.331728 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.332345 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:52.332359 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:52.334047 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:52.336091 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.336096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.336098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.336351 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.336715 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.336722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.336726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.336822 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.337263 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:52.337269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:52.337368 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 02:48:52.337512 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 02:48:52.337630 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":21}
I20260927 02:48:52.338220 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.328511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.61:43649)
0927 02:48:52.328639 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.328646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.328663 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:48:52.328710 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.328713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.328762 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.328865 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.329402 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.330112 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.331204 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.331491 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.331597 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.334183 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.334187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.334190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.334218 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.335985 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.336525 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.336538 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.336542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.336631 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.336934 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.336938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.336941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.337159 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.337258 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.337570 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.337688 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:48:52.338599 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.328763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.0.98.62:36901)
0927 02:48:52.328896 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.328900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.328915 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:48:52.328964 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.328967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.329004 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.329105 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.329847 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.330552 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.331482 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.331788 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.331930 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.334932 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.334935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.334937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.334957 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.336618 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.337092 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.337095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.337097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.337168 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.337471 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.337487 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.337490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.337698 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.337913 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.338221 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.338331 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:48:52.338919 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.329353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60082)
0927 02:48:52.329498 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.329522 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:52.329585 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:52.329910 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:52.329913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:52.330090 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:52.330176 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.330182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.331162 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.331166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.332059 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.332074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.332764 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.332770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.332895 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.333351 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:52.333362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:52.335223 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:52.337376 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.337378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.337380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.337620 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.337917 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.337921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.337925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.338013 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.338445 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:52.338448 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:52.338547 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 02:48:52.338625 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:48:52.338775 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 02:48:52.338981 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:52.339411 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.329437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.0.98.60:37469)
0927 02:48:52.329712 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.329716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.329727 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:48:52.329755 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.329757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.329790 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.329888 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.330292 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.331041 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.332219 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.332615 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.333101 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.335374 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.335378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.335380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.335406 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.337204 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.337730 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.337734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.337736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.337807 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.338128 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.338130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.338132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.338338 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.339076 (+ 738us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.339153 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.339275 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":214}
I20260927 02:48:52.340971 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } attempt: 3
I20260927 02:48:52.342607 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe: Bootstrap starting.
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:48:52.345948 22641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:52.346881 22641 log.cc:824] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:48:52 dist-test-slave-gkrl krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:48:52.347817 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.339393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60098)
0927 02:48:52.339975 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.339984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:52.340081 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:52.340272 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:52.340275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:52.340428 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:52.340480 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.340485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.341219 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.341225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.341962 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.341972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.342595 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.342598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.342710 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.343137 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:52.343153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:52.344487 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:52.346371 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.346373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.346375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.346597 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.346857 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.346862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.346864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.346941 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.347354 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:52.347356 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:52.347435 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 02:48:52.347550 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 02:48:52.347669 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":21}
I20260927 02:48:52.348387 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.339721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.0.98.60:37469)
0927 02:48:52.339864 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.339868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.340048 (+ 180us) server_negotiation.cc:407] Connection header received
0927 02:48:52.340148 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.340151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.340182 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.340264 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.340586 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.341106 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.342075 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.342404 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.342492 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.344631 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.344635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.344637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.344657 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.346270 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.346687 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.346693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.346695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.346770 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.347046 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.347049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.347051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.347217 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.347320 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.348110 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.348218 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 02:48:52.349170 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe: No bootstrap required, opened a new log
I20260927 02:48:52.351279 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.339725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.98.61:43649)
0927 02:48:52.339823 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.339826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.339835 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:52.339861 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.339863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.339979 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.340062 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.340855 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.341537 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.342543 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.342825 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.342910 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.345747 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.345751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.345753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.345779 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.349206 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.349692 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.349697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.349701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.349791 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.350102 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.350108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.350113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.350312 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.350505 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.351005 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.351137 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:48:52.352313 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.341292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37469 (local address 127.0.0.1:60106)
0927 02:48:52.341427 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:52.341440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:52.341502 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:52.342221 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:52.342224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:52.342388 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:52.342488 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.342494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.344137 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.344141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.344971 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:52.345006 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.345874 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.345878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:52.345964 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.346402 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:52.346416 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:52.347790 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:52.350398 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.350402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.350406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.350643 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.350938 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:52.350940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:52.350942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:52.351039 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:52.351782 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:52.351785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:52.351902 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 02:48:52.352034 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 02:48:52.352155 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 02:48:52.352744 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.341682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.0.98.60:37469)
0927 02:48:52.341995 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.342006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.342020 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:48:52.342067 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.342070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.342107 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.342586 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.342735 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.344001 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.345275 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.345734 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.346271 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.347948 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.347952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.347955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.347978 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.350289 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.350750 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.350753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.350754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.350840 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.351142 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.351146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.351147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.351331 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.351422 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.352482 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.352580 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260927 02:48:52.354097 22791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec: Bootstrap starting.
I20260927 02:48:52.354874 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:52.355221 22716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c: Bootstrap starting.
I20260927 02:48:52.356245 22615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f1199f443c734667a2d845d8671906fe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60002
I20260927 02:48:52.356343 22791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:52.357131 22791 log.cc:824] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:52.357323 22716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:52.358091 22716 log.cc:824] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:52.359097 22791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec: No bootstrap required, opened a new log
I20260927 02:48:52.358943 22641 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } }
I20260927 02:48:52.359175 22641 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1199f443c734667a2d845d8671906fe, State: Initialized, Role: FOLLOWER
I20260927 02:48:52.359496 22641 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } }
I20260927 02:48:52.359956 22716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c: No bootstrap required, opened a new log
I20260927 02:48:52.360114 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.360407 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:52.360754 22641 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:52.360821 22615 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:48:52.360929 22615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:52.362109 22615 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } }
I20260927 02:48:52.362984 22615 leader_election.cc:288] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [CANDIDATE]: Term 1 election: Requested vote from peers 3d0fbfb98d1b4c349fe3e650f15c803c (127.0.98.61:43649), 088045bd960048e39f54f6fdd0acd6ec (127.0.98.60:37469)
I20260927 02:48:52.363150 22791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } }
I20260927 02:48:52.363360 22791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088045bd960048e39f54f6fdd0acd6ec, State: Initialized, Role: FOLLOWER
I20260927 02:48:52.363648 22791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } }
I20260927 02:48:52.364852 22791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:52.364882 22765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1199f443c734667a2d845d8671906fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "088045bd960048e39f54f6fdd0acd6ec"
I20260927 02:48:52.365061 22765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:52.366933 22765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1199f443c734667a2d845d8671906fe in term 1.
W20260927 02:48:52.369174 22811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:52.369252 22811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:52.365690 22573 master_runner.cc:186] Error getting master registration for 127.0.98.61:43649: OK, instance_id { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" instance_seqno: 1790477332087385 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:48:52.369516 22573 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:48:52.369709 22591 leader_election.cc:302] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088045bd960048e39f54f6fdd0acd6ec, f1199f443c734667a2d845d8671906fe; no voters:
I20260927 02:48:52.369966 22800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:52.370167 22800 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [term 1 LEADER]: Becoming Leader. State: Replica: f1199f443c734667a2d845d8671906fe, State: Running, Role: LEADER
I20260927 02:48:52.370465 22800 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } }
W20260927 02:48:52.370849 22827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:52.370921 22827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:52.373160 22716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } }
I20260927 02:48:52.373317 22716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0fbfb98d1b4c349fe3e650f15c803c, State: Initialized, Role: FOLLOWER
I20260927 02:48:52.373600 22716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } }
I20260927 02:48:52.374001 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.374120 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:52.374667 22716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:52.374737 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1199f443c734667a2d845d8671906fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c"
I20260927 02:48:52.374944 22690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:52.376214 22690 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1199f443c734667a2d845d8671906fe in term 1.
I20260927 02:48:52.376683 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1199f443c734667a2d845d8671906fe. Latest consensus state: current_term: 1 leader_uuid: "f1199f443c734667a2d845d8671906fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.377130 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:52.377548 22839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:52.378521 22839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:52.381589 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.381708 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:48:52.381914 22844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:52.382001 22844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:52.398489 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.380832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.0.98.61:43649)
0927 02:48:52.381240 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.381246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.382287 (+ 1041us) server_negotiation.cc:407] Connection header received
0927 02:48:52.382459 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.382463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.382517 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.382566 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.385949 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.386740 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.389783 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.390118 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.390189 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.392882 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.392885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.392888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.392912 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.394569 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.395033 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.395037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.395038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.395115 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.395385 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.395388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.395390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.395589 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.395707 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.398228 (+ 2521us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.398339 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260927 02:48:52.401405 22765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [term 1 FOLLOWER]: Refusing update from remote peer f1199f443c734667a2d845d8671906fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:52.401620 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.381162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.0.98.62:36901)
0927 02:48:52.381259 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.381263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.384798 (+ 3535us) server_negotiation.cc:407] Connection header received
0927 02:48:52.384951 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.384954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.384993 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.385041 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.387939 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.388677 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.390674 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.391006 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.391075 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.396037 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.396040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.396045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.396070 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.397693 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.400099 (+ 2406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.400102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.400104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.400211 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.400701 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.400705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.400707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.400923 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.401034 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.401362 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.401474 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:48:52.402112 22830 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:52.406203 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1199f443c734667a2d845d8671906fe. Latest consensus state: current_term: 1 leader_uuid: "f1199f443c734667a2d845d8671906fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.406323 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:52.408123 22690 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [term 1 FOLLOWER]: Refusing update from remote peer f1199f443c734667a2d845d8671906fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:52.408636 22830 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:52.411921 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1199f443c734667a2d845d8671906fe. Latest consensus state: current_term: 1 leader_uuid: "f1199f443c734667a2d845d8671906fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.412048 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:52.416383 22848 mvcc.cc:205] Tried to move back new op lower bound from 7333795153500430336 to 7333795153392660480. Current Snapshot: MvccSnapshot[applied={T|T < 7333795153500430336}]
I20260927 02:48:52.416646 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1199f443c734667a2d845d8671906fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.416752 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:52.417748 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1199f443c734667a2d845d8671906fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.417866 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:52.418018 22850 mvcc.cc:205] Tried to move back new op lower bound from 7333795153500430336 to 7333795153392660480. Current Snapshot: MvccSnapshot[applied={T|T < 7333795153500430336}]
I20260927 02:48:52.422474 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:52.381512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.0.98.60:37469)
0927 02:48:52.381627 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.381632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.399950 (+ 18318us) server_negotiation.cc:407] Connection header received
0927 02:48:52.400077 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.400083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.400127 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.400201 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.406443 (+ 6242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.407252 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.413175 (+ 5923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:52.413547 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:52.413619 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:52.418740 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:52.418744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:52.418747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:52.418772 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:52.420486 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.421182 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.421185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.421187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.421268 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:52.421562 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:52.421565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:52.421567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:52.421763 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:52.421870 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:52.422211 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 02:48:52.422327 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:48:52.423267 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1199f443c734667a2d845d8671906fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.423440 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:52.424892 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1199f443c734667a2d845d8671906fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1199f443c734667a2d845d8671906fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } peers { permanent_uuid: "3d0fbfb98d1b4c349fe3e650f15c803c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43649 } } peers { permanent_uuid: "088045bd960048e39f54f6fdd0acd6ec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37469 } } }
I20260927 02:48:52.425019 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:52.425691 22852 mvcc.cc:205] Tried to move back new op lower bound from 7333795153500430336 to 7333795153392660480. Current Snapshot: MvccSnapshot[applied={T|T < 7333795153500430336}]
I20260927 02:48:52.429984 22839 catalog_manager.cc:1347] Generated new cluster ID: 8b7c9743fe0242b8820111dfdc8e48d6
I20260927 02:48:52.430056 22839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:52.455974 22839 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:52.457175 22839 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:52.471695 22839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1199f443c734667a2d845d8671906fe: Generated new TSK 0
I20260927 02:48:52.472534 22839 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:48:52.504211 22644 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.501189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.0.98.62:36901)
0927 02:48:52.501301 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.501305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.503089 (+ 1784us) server_negotiation.cc:407] Connection header received
0927 02:48:52.503341 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.503344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.503393 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.503448 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.504080 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60022: BlockingRecv error: recv got EOF from 127.0.0.1:60022 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 02:48:52.505854 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.501855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.98.60:37469)
0927 02:48:52.501962 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.501966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.504583 (+ 2617us) server_negotiation.cc:407] Connection header received
0927 02:48:52.505096 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.505099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.505148 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.505201 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.505739 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60130: BlockingRecv error: recv got EOF from 127.0.0.1:60130 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 02:48:52.506897 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.502532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.98.61:43649)
0927 02:48:52.502641 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.502645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.504996 (+ 2351us) server_negotiation.cc:407] Connection header received
0927 02:48:52.506224 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.506228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.506276 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.506326 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.506792 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33690: BlockingRecv error: recv got EOF from 127.0.0.1:33690 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 02:48:52.511317 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.509893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.0.98.61:43649)
0927 02:48:52.509992 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.509996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.510431 (+ 435us) server_negotiation.cc:407] Connection header received
0927 02:48:52.510633 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.510636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.510705 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.510759 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.511227 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33692: BlockingRecv error: recv got EOF from 127.0.0.1:33692 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:48:52.516144 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.514634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.98.61:43649)
0927 02:48:52.514736 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.514739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.515253 (+ 514us) server_negotiation.cc:407] Connection header received
0927 02:48:52.515431 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.515435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.515481 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.515531 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.516054 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33708: BlockingRecv error: recv got EOF from 127.0.0.1:33708 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 02:48:52.521513 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.520129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.0.98.61:43649)
0927 02:48:52.520217 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.520221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.520685 (+ 464us) server_negotiation.cc:407] Connection header received
0927 02:48:52.520856 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.520859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.520903 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.520952 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.521423 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33716: BlockingRecv error: recv got EOF from 127.0.0.1:33716 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:48:52.528970 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.527506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.0.98.61:43649)
0927 02:48:52.527635 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.527639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.528153 (+ 514us) server_negotiation.cc:407] Connection header received
0927 02:48:52.528329 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.528332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.528383 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.528431 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.528887 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33724: BlockingRecv error: recv got EOF from 127.0.0.1:33724 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 02:48:52.540683 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.539089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.98.61:43649)
0927 02:48:52.539192 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.539196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.539694 (+ 498us) server_negotiation.cc:407] Connection header received
0927 02:48:52.539932 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.539935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.539986 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.540057 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.540576 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33734: BlockingRecv error: recv got EOF from 127.0.0.1:33734 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:48:52.560611 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.558946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.0.98.61:43649)
0927 02:48:52.559066 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.559071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.559675 (+ 604us) server_negotiation.cc:407] Connection header received
0927 02:48:52.559858 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.559862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.559935 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.559986 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.560505 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33740: BlockingRecv error: recv got EOF from 127.0.0.1:33740 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 02:48:52.596393 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.595064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.0.98.61:43649)
0927 02:48:52.595188 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.595193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.595435 (+ 242us) server_negotiation.cc:407] Connection header received
0927 02:48:52.595607 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.595610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.595668 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.595757 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.596264 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33752: BlockingRecv error: recv got EOF from 127.0.0.1:33752 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 02:48:52.663609 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.662501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.0.98.61:43649)
0927 02:48:52.662650 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.662654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.662809 (+ 155us) server_negotiation.cc:407] Connection header received
0927 02:48:52.662912 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.662915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.662961 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.663048 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.663457 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33760: BlockingRecv error: recv got EOF from 127.0.0.1:33760 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 02:48:52.794799 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:52.793778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.0.98.61:43649)
0927 02:48:52.793917 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:52.793921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:52.794014 (+ 93us) server_negotiation.cc:407] Connection header received
0927 02:48:52.794143 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:52.794146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:52.794192 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:52.794276 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:52.794707 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33762: BlockingRecv error: recv got EOF from 127.0.0.1:33762 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 02:48:53.053808 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:53.052913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.0.98.61:43649)
0927 02:48:53.053053 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:53.053058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:53.053076 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:48:53.053193 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:53.053196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:53.053247 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:53.053315 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:53.053715 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33774: BlockingRecv error: recv got EOF from 127.0.0.1:33774 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:48:53.372177 22827 catalog_manager.cc:1259] Loaded cluster ID: 8b7c9743fe0242b8820111dfdc8e48d6
I20260927 02:48:53.372262 22827 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec: loading cluster ID for follower catalog manager: success
I20260927 02:48:53.373891 22827 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec: acquiring CA information for follower catalog manager: success
I20260927 02:48:53.375105 22827 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 088045bd960048e39f54f6fdd0acd6ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:48:53.383020 22844 catalog_manager.cc:1259] Loaded cluster ID: 8b7c9743fe0242b8820111dfdc8e48d6
I20260927 02:48:53.383080 22844 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c: loading cluster ID for follower catalog manager: success
I20260927 02:48:53.384644 22844 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c: acquiring CA information for follower catalog manager: success
I20260927 02:48:53.385679 22844 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d0fbfb98d1b4c349fe3e650f15c803c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:48:53.571954 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:53.568241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.98.61:43649)
0927 02:48:53.568487 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:53.568494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:53.568644 (+ 150us) server_negotiation.cc:407] Connection header received
0927 02:48:53.568748 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:53.568751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:53.568870 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:53.568965 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:53.569202 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.569727 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.570517 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.570740 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.570816 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:53.571081 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:53.571513 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:53.571606 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:53.571684 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:48:53.571731 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
I20260927 02:48:53.578397 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:53.574886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.0.98.60:37469)
0927 02:48:53.575117 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:53.575121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:53.575135 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:48:53.575319 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:53.575321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:53.575444 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:53.575536 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:53.575703 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.576405 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.577146 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.577358 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.577431 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:53.577710 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:53.578088 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:53.578178 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:53.578239 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:48:53.578282 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260927 02:48:53.585641 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:53.581299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.0.98.61:43649)
0927 02:48:53.581524 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:53.581528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:53.582074 (+ 546us) server_negotiation.cc:407] Connection header received
0927 02:48:53.582148 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:53.582151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:53.582210 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:53.582277 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:53.582598 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.583322 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.584462 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.584798 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.584893 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:53.585176 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:53.585239 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:53.585322 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:53.585422 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:48:53.585469 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 02:48:53.586050 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:53.581299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.0.98.62:36901)
0927 02:48:53.581546 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:53.581552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:53.581723 (+ 171us) server_negotiation.cc:407] Connection header received
0927 02:48:53.581884 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:53.581887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:53.582059 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:53.582211 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:53.582406 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.583209 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.584429 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.584689 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.584802 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:53.585132 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:53.585432 (+ 300us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:53.585795 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:53.585829 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 02:48:53.585884 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260927 02:48:53.586932 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:53.581606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.0.98.60:37469)
0927 02:48:53.581726 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:53.581731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:53.582219 (+ 488us) server_negotiation.cc:407] Connection header received
0927 02:48:53.582265 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:53.582270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:53.582312 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:53.582381 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:53.583652 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.584213 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.585339 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:53.585900 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:53.586000 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:53.586364 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:53.586428 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:53.586512 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:53.586762 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:48:53.586812 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:48:53.589092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22573
I20260927 02:48:53.595538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22648
I20260927 02:48:53.601318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22722
2026-09-27T02:48:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3075 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:53 dist-test-slave-gkrl krb5kdc[22939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:53 dist-test-slave-gkrl krb5kdc[22939](info): set up 2 sockets
Sep 27 02:48:53 dist-test-slave-gkrl krb5kdc[22939](info): commencing operation
krb5kdc: starting...
W20260927 02:48:54.640195 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:54 dist-test-slave-gkrl krb5kdc[22939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:54.756057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37251
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37251,127.0.98.61:37361,127.0.98.60:36431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:54.860357 22952 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:48:54.860622 22952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:54.860646 22952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:54.860680 22952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:54.864586 22952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:54.864636 22952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:54.864655 22952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:54.864671 22952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:54.864686 22952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:54.869633 22952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37251,127.0.98.61:37361,127.0.98.60:36431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:54.870666 22952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:54.871559 22952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:54.879732 22960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:54.879740 22958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:54.879745 22957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:54.879995 22952 server_base.cc:1034] running on GCE node
I20260927 02:48:54.880615 22952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:54.881642 22952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:54.882848 22952 hybrid_clock.cc:648] HybridClock initialized: now 1790477334882830 us; error 28 us; skew 500 ppm
Sep 27 02:48:54 dist-test-slave-gkrl krb5kdc[22939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:54.885461 22952 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:54.886305 22952 webserver.cc:460] Webserver started at http://127.0.98.62:45031/ using document root <none> and password file <none>
I20260927 02:48:54.886833 22952 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:54.886924 22952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:54.887161 22952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:54.889146 22952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/data/instance:
uuid: "1fe3b5d82e5d43b894dc2104be38a58b"
format_stamp: "Formatted at 2026-09-27 02:48:54 on dist-test-slave-gkrl"
server_key: "fcbd9203cf674291d1e61c55922a82de"
server_key_iv: "cd4d4da22a5a26383eca30fa2488e574"
server_key_version: "encryptionkey@0"
I20260927 02:48:54.889691 22952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fe3b5d82e5d43b894dc2104be38a58b"
format_stamp: "Formatted at 2026-09-27 02:48:54 on dist-test-slave-gkrl"
server_key: "fcbd9203cf674291d1e61c55922a82de"
server_key_iv: "cd4d4da22a5a26383eca30fa2488e574"
server_key_version: "encryptionkey@0"
I20260927 02:48:54.893186 22952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:48:54.895480 22967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:54.896670 22952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:54.896804 22952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/wal
uuid: "1fe3b5d82e5d43b894dc2104be38a58b"
format_stamp: "Formatted at 2026-09-27 02:48:54 on dist-test-slave-gkrl"
server_key: "fcbd9203cf674291d1e61c55922a82de"
server_key_iv: "cd4d4da22a5a26383eca30fa2488e574"
server_key_version: "encryptionkey@0"
I20260927 02:48:54.896909 22952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:54.923678 22952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:54.926704 22952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:54.926909 22952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:54.934159 22952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37251
I20260927 02:48:54.934481 23019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37251 every 8 connection(s)
I20260927 02:48:54.935133 22952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:54.937956 23020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:54.941457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 22952
I20260927 02:48:54.941617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:54.941905 392 external_mini_cluster.cc:1428] Setting key d697b829e54d68bbfbcc367fb800a8f4
I20260927 02:48:54.941810 23020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 02:48:54 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:54.958348 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:54.944151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.0.98.62:37251)
0927 02:48:54.944733 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:54.944746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:54.944792 (+ 46us) server_negotiation.cc:407] Connection header received
0927 02:48:54.945890 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:54.945895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:54.946085 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:54.946287 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:54.947127 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:54.948043 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:54.949009 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:54.949431 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:54.949633 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:54.953134 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:54.953140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:54.953150 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:54.953174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:54.955160 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:54.955794 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:54.955799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:54.955800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:54.955990 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:54.956397 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:54.956399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:54.956401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:54.956775 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:54.956889 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:54.957377 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 02:48:54.957580 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":453,"spinlock_wait_cycles":24448,"thread_start_us":177,"threads_started":1}
I20260927 02:48:54.958349 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:54.944150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37251 (local address 127.0.0.1:46242)
0927 02:48:54.944627 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:54.944678 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:54.945543 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:54.946284 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:54.946290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:54.946745 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:54.946981 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:54.946994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:54.948160 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:54.948164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:54.948887 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:54.948897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:54.949587 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:54.949591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:54.949744 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:54.950421 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:54.950447 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:54.952958 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:54.955292 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:54.955304 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:54.955317 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:54.955655 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:54.956098 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:54.956104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:54.956109 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:54.956266 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:54.956910 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:54.956919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:54.957106 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 02:48:54.957321 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 02:48:54.957671 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":63,"thread_start_us":136,"threads_started":1}
I20260927 02:48:54.960865 23020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:54.961552 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:54.961312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41250)
0927 02:48:54.961423 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:54.961493 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:48:54.961818 22971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
I20260927 02:48:54.962708 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:54.962441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41262)
0927 02:48:54.962571 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:54.962657 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:48:54.962873 23020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37361: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:48:54.968892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37361
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37251,127.0.98.61:37361,127.0.98.60:36431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:48:55.012122 23020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attempt: 1
I20260927 02:48:55.013046 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.012732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41274)
0927 02:48:55.012902 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.012966 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 02:48:55.013789 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.013561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41276)
0927 02:48:55.013678 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.013731 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 02:48:55.013917 23020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37361: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
W20260927 02:48:55.076362 23027 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:48:55.076668 23027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:55.076740 23027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:55.076821 23027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:55.081744 23027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:55.081821 23027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:55.081859 23027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:55.081895 23027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:55.081938 23027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:55.082073 23020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attempt: 2
I20260927 02:48:55.082846 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.082553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41292)
0927 02:48:55.082707 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.082775 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 02:48:55.083510 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.083300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41300)
0927 02:48:55.083404 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.083450 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:48:55.083658 23020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37361: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37361: connect: Connection refused (error 111)
I20260927 02:48:55.087345 23027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37251,127.0.98.61:37361,127.0.98.60:36431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:55.088438 23027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:55.089412 23027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:55.097637 23033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:55.097688 23032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:55.097640 23035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:55.098016 23027 server_base.cc:1034] running on GCE node
I20260927 02:48:55.098546 23027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:55.099697 23027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:55.100907 23027 hybrid_clock.cc:648] HybridClock initialized: now 1790477335100905 us; error 50 us; skew 500 ppm
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:55.103557 23027 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:48:55.104478 23027 webserver.cc:460] Webserver started at http://127.0.98.61:45343/ using document root <none> and password file <none>
I20260927 02:48:55.105048 23027 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:55.105144 23027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:55.105377 23027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:55.107452 23027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/data/instance:
uuid: "e62e08efb0704e05967763453cbb1f21"
format_stamp: "Formatted at 2026-09-27 02:48:55 on dist-test-slave-gkrl"
server_key: "ff51d82d6643a5d57897825bb91aef01"
server_key_iv: "337d7c0eb75b19b0cc7b7c916b4596f0"
server_key_version: "encryptionkey@0"
I20260927 02:48:55.108071 23027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/wal/instance:
uuid: "e62e08efb0704e05967763453cbb1f21"
format_stamp: "Formatted at 2026-09-27 02:48:55 on dist-test-slave-gkrl"
server_key: "ff51d82d6643a5d57897825bb91aef01"
server_key_iv: "337d7c0eb75b19b0cc7b7c916b4596f0"
server_key_version: "encryptionkey@0"
I20260927 02:48:55.111653 23027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:48:55.114295 23042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:55.115484 23027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:55.115631 23027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/wal
uuid: "e62e08efb0704e05967763453cbb1f21"
format_stamp: "Formatted at 2026-09-27 02:48:55 on dist-test-slave-gkrl"
server_key: "ff51d82d6643a5d57897825bb91aef01"
server_key_iv: "337d7c0eb75b19b0cc7b7c916b4596f0"
server_key_version: "encryptionkey@0"
I20260927 02:48:55.115744 23027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:55.130270 23027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:55.133358 23027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:55.133560 23027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:55.140332 23027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37361
I20260927 02:48:55.140602 23094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37361 every 8 connection(s)
I20260927 02:48:55.141314 23027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:48:55.144115 23095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:55.144846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 23027
I20260927 02:48:55.144989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-1/wal/instance
I20260927 02:48:55.145321 392 external_mini_cluster.cc:1428] Setting key d57bf2074c698fff52bda8719330c52b
I20260927 02:48:55.148638 23095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:55.163007 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.150617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.0.98.62:37251)
0927 02:48:55.150776 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.150781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.151618 (+ 837us) server_negotiation.cc:407] Connection header received
0927 02:48:55.152594 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.152601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.152668 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.152780 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.153934 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.154548 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.155760 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.156156 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.156243 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.159051 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.159058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.159064 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.159097 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.160903 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.161391 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.161398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.161402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.161507 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.161826 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.161830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.161833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.162064 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.162177 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.162672 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.162811 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:48:55.163651 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.150969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37251 (local address 127.0.0.1:46248)
0927 02:48:55.151445 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.151495 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:55.152378 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:55.152935 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:55.152942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:55.153343 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:55.153797 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.153809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.154667 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.154670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.155632 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.155642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.156320 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.156328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.156568 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.157199 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:55.157225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:55.158885 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:55.161020 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.161025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.161036 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.161288 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.161608 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.161611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.161612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.161733 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.162170 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:55.162184 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:55.162421 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 02:48:55.162570 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 02:48:55.162805 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260927 02:48:55.165031 23095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:48:55.172848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8LsHFz/build/debug/bin/kudu
/tmp/dist-test-task8LsHFz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36431
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37251,127.0.98.61:37361,127.0.98.60:36431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:48:55.174301 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.165537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41308)
0927 02:48:55.165664 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.165676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:55.165736 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:55.166338 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:55.166340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:55.166464 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:55.166540 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.166545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.167564 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.167567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.168279 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.168288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.168922 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.168925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.169010 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.169448 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:55.169459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:55.170729 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:55.172642 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.172648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.172653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.172892 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.173233 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.173238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.173243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.173346 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.173832 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:55.173838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:55.173957 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 02:48:55.174086 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 02:48:55.174195 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:48:55.174595 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.165607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.0.98.61:37361)
0927 02:48:55.165921 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.165932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.165952 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:48:55.166000 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.166003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.166218 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.166364 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.166641 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.167420 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.168388 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.168810 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.168925 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.170877 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.170886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.170895 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.170915 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.172528 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.173028 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.173036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.173038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.173129 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.173465 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.173468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.173469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.173720 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.173831 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.174218 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.174378 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":225,"thread_start_us":157,"threads_started":1}
I20260927 02:48:55.174810 23020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attempt: 3
I20260927 02:48:55.176522 23095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:55.177210 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.176979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59738)
0927 02:48:55.177082 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.177159 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:48:55.177398 23045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
I20260927 02:48:55.178702 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.178000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59740)
0927 02:48:55.178400 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.178455 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":100,"threads_started":1}
W20260927 02:48:55.178920 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36431: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:48:55.184917 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.175227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41324)
0927 02:48:55.175507 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.175524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:55.175606 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:55.176089 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:55.176092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:55.176270 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:55.176796 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.176810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.177961 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.177968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.178982 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.179006 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.179738 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.179741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.179824 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.180342 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:55.180357 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:55.181536 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:55.183503 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.183505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.183507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.183704 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.184029 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.184031 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.184032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.184114 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.184548 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:55.184550 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:55.184632 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 02:48:55.184713 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:48:55.184810 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260927 02:48:55.185181 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.175684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.0.98.61:37361)
0927 02:48:55.175805 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.175811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.175825 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:48:55.175911 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.175916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.175966 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.176045 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.176922 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.177797 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.179163 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.179619 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.179745 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.181672 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.181678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.181683 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.181714 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.183384 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.183804 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.183810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.183813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.183934 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.184210 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.184216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.184220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.184443 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.184552 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.184785 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.184889 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:48:55.185858 23020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:55.186766 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.186506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59748)
0927 02:48:55.186661 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.186708 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 02:48:55.187431 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.187195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59764)
0927 02:48:55.187332 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.187381 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 02:48:55.187573 23020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36431: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
I20260927 02:48:55.218737 23020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } attempt: 1
I20260927 02:48:55.219659 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.219355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59776)
0927 02:48:55.219513 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.219574 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 02:48:55.220511 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.220292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59788)
0927 02:48:55.220404 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.220459 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:48:55.220652 23020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36431: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
I20260927 02:48:55.228173 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } attempt: 1
I20260927 02:48:55.228878 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.228622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59798)
0927 02:48:55.228753 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.228814 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 02:48:55.229548 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.229347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59802)
0927 02:48:55.229448 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.229494 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:48:55.229698 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36431: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
W20260927 02:48:55.284538 23101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:48:55.284773 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:55.284830 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:55.284905 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:55.288767 23101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:55.288812 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:55.288831 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:55.288875 23101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:55.288909 23101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:55.293879 23101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37251,127.0.98.61:37361,127.0.98.60:36431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:48:55.294778 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:55.295688 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:48:55.295815 23020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } attempt: 2
I20260927 02:48:55.296617 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.296330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59818)
0927 02:48:55.296465 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.296526 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 02:48:55.297372 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.297160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59826)
0927 02:48:55.297268 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.297313 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:48:55.297549 23020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36431: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
I20260927 02:48:55.297881 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } attempt: 2
I20260927 02:48:55.298507 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.298240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59830)
0927 02:48:55.298359 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.298432 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 02:48:55.299196 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.299006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59832)
0927 02:48:55.299101 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.299146 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:48:55.299347 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36431: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36431: connect: Connection refused (error 111)
W20260927 02:48:55.304301 23109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:55.304317 23108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:55.304384 23101 server_base.cc:1034] running on GCE node
W20260927 02:48:55.304306 23111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:55.305045 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:55.306054 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:55.307222 23101 hybrid_clock.cc:648] HybridClock initialized: now 1790477335307210 us; error 22 us; skew 500 ppm
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:55.309863 23101 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:48:55.310732 23101 webserver.cc:460] Webserver started at http://127.0.98.60:45193/ using document root <none> and password file <none>
I20260927 02:48:55.311540 23101 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:55.311652 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:55.311935 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:55.314047 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/data/instance:
uuid: "fac6a2fe388d4200a13e908c1f39738f"
format_stamp: "Formatted at 2026-09-27 02:48:55 on dist-test-slave-gkrl"
server_key: "ee1e28a8a9ec86d0345048679cbd3787"
server_key_iv: "e7be94456fec6ed25b3de787be93cda7"
server_key_version: "encryptionkey@0"
I20260927 02:48:55.314603 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/wal/instance:
uuid: "fac6a2fe388d4200a13e908c1f39738f"
format_stamp: "Formatted at 2026-09-27 02:48:55 on dist-test-slave-gkrl"
server_key: "ee1e28a8a9ec86d0345048679cbd3787"
server_key_iv: "e7be94456fec6ed25b3de787be93cda7"
server_key_version: "encryptionkey@0"
I20260927 02:48:55.318122 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:48:55.320613 23118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:55.321826 23101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:48:55.321990 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/wal
uuid: "fac6a2fe388d4200a13e908c1f39738f"
format_stamp: "Formatted at 2026-09-27 02:48:55 on dist-test-slave-gkrl"
server_key: "ee1e28a8a9ec86d0345048679cbd3787"
server_key_iv: "e7be94456fec6ed25b3de787be93cda7"
server_key_version: "encryptionkey@0"
I20260927 02:48:55.322101 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:55.353609 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:55.356377 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:55.356594 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:55.363363 23101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36431
I20260927 02:48:55.363648 23170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36431 every 8 connection(s)
I20260927 02:48:55.364334 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:48:55.366940 23171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:55.369598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8LsHFz/build/debug/bin/kudu as pid 23101
I20260927 02:48:55.369738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8LsHFz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477031916752-392-0/minicluster-data/master-2/wal/instance
I20260927 02:48:55.370074 392 external_mini_cluster.cc:1428] Setting key c434028283c6acfa1e7a624db6971dad
I20260927 02:48:55.371439 23171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:48:55.383641 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.373210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.0.98.62:37251)
0927 02:48:55.373364 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.373369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.373787 (+ 418us) server_negotiation.cc:407] Connection header received
0927 02:48:55.373917 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.373920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.373963 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.374038 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.374761 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.375496 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.376862 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.377189 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.377279 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.379786 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.379791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.379794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.379822 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.381635 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.382186 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.382189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.382191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.382290 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.382631 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.382636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.382637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.382826 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.382934 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.383369 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.383476 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 02:48:55.384526 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.373086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37251 (local address 127.0.0.1:46256)
0927 02:48:55.373631 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.373687 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:55.373821 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:55.374122 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:55.374129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:55.374449 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:55.374620 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.374630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.375627 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.375631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.376745 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.376754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.377325 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.377329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.377456 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.377940 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:55.377963 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:55.379598 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:55.381741 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.381748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.381755 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.382087 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.382391 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.382396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.382398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.382533 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.382934 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:55.382942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:55.383166 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 02:48:55.383327 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 02:48:55.383606 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":120,"threads_started":1}
I20260927 02:48:55.384531 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.371334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.0.98.60:36431)
0927 02:48:55.371704 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.371743 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.371788 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:48:55.372469 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.372495 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.372880 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.373210 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.374168 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.375417 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.376643 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.377099 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.377266 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.380355 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.380363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.380376 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.380401 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.382175 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.382568 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.382574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.382576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.382659 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.382939 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.382942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.382944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.383185 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.383378 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.383615 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.383907 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
I20260927 02:48:55.385733 23171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:48:55.390527 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } attempt: 3
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:55.394848 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.386175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41338)
0927 02:48:55.386254 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.386271 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:55.386334 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:55.386660 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:55.386666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:55.386833 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:55.386938 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.386945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.387974 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.387978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.388782 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.388790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.389313 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.389315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.389382 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.389859 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:55.389872 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:55.391325 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:55.393327 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.393330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.393332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.393572 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.393871 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.393873 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.393874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.393938 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.394380 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:55.394382 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:55.394487 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 02:48:55.394593 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 02:48:55.394706 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:48:55.394881 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.386292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.0.98.61:37361)
0927 02:48:55.386429 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.386433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.386446 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:48:55.386489 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.386491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.386537 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.386649 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.387066 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.387844 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.388890 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.389211 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.389310 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.391511 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.391516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.391519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.391541 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.393210 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.393684 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.393690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.393695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.393773 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.394037 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.394042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.394044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.394246 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.394337 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.394653 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.394751 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 02:48:55.395668 23171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:48:55.396343 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.386833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.0.98.62:37251)
0927 02:48:55.386936 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.386942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.386956 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:48:55.387098 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.387101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.387136 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.387193 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.387861 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.388638 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.389611 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.389823 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.389922 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.392504 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.392508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.392510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.392533 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.394147 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.394825 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.394829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.394830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.395080 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.395410 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.395413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.395415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.395643 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.395840 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.396057 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.396169 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:48:55.396725 23020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } attempt: 3
I20260927 02:48:55.402293 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.390936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.0.98.60:36431)
0927 02:48:55.391038 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.391042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.391148 (+ 106us) server_negotiation.cc:407] Connection header received
0927 02:48:55.391240 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.391246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.391288 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.391355 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.391846 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.392596 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.393730 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.394124 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.394241 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.397913 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.397916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.397918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.397939 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.399650 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.400133 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.400136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.400138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.400215 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.400587 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.400590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.400592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.400780 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.400866 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.402001 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.402124 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:48:55.402580 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.390903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59850)
0927 02:48:55.391059 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.391079 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:55.391170 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:55.391367 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:55.391370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:55.391576 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:55.391714 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.391724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.392710 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.392713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.393607 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.393619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.394256 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.394260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.394371 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.394852 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:55.394870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:55.397772 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:55.399753 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.399757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.399759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.400026 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.400353 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.400356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.400358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.400493 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.400912 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:55.400915 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:55.401013 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 02:48:55.401112 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:48:55.401248 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:48:55 dist-test-slave-gkrl krb5kdc[22939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:48:55.406078 23095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21: Bootstrap starting.
I20260927 02:48:55.407964 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.397787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.0.98.61:37361)
0927 02:48:55.397896 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.397899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.397950 (+ 51us) server_negotiation.cc:407] Connection header received
0927 02:48:55.398133 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.398136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.398176 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.398233 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.399101 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.399761 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.400843 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.401178 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.401241 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.404307 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.404310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.404313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.404338 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.405991 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.406497 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.406502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.406503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.406590 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.406931 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.406934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.406935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.407149 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.407304 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.407683 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.407806 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:48:55.408481 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.396910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59866)
0927 02:48:55.397003 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.397015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:55.397094 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:55.398061 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:55.398066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:55.398259 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:55.398366 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.398377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.399589 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.399595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.400414 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.400423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.401069 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.401075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.401166 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.401591 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:55.401605 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:55.404503 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:55.407041 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.407044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.407046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.407276 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.407566 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.407570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.407571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.407655 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.408072 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:55.408075 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:55.408168 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 02:48:55.408256 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 02:48:55.408366 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:48:55.408821 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.397034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36431 (local address 127.0.0.1:59882)
0927 02:48:55.397406 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:55.397419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:55.397491 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:55.397731 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:55.397734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:55.397896 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:55.398008 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.398014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.399673 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.399676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.400493 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:55.400503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.401141 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.401145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:55.401227 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.401780 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:55.401791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:55.404080 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:55.407005 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.407008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.407010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.407241 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.407619 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:55.407622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:55.407624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:55.407708 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:55.408372 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:55.408375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:55.408464 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 02:48:55.408556 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:48:55.408673 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260927 02:48:55.409123 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.397312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.0.98.60:36431)
0927 02:48:55.397867 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.397875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.397887 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:48:55.397912 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.397915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.397952 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.398478 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.398601 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.399472 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.400563 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.400960 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.401354 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.404628 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.404631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.404633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.404661 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.406945 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.407377 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.407380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.407381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.407470 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.407750 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.407753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.407758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.407966 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.408756 (+ 790us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.408815 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.408945 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":481,"thread_start_us":95,"threads_started":1}
I20260927 02:48:55.409402 23095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:55.410259 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.397146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.98.60:36431)
0927 02:48:55.397441 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.397452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.397480 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:48:55.397565 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.397569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.397606 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.398118 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.398222 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.399559 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.400631 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.401022 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.401405 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.404232 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.404235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.404238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.404260 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.406883 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.407371 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.407375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.407378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.407514 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.407817 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.407819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.407821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.408240 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.409913 (+ 1673us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.409981 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.410081 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
I20260927 02:48:55.410351 23095 log.cc:824] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:55.411795 23171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f: Bootstrap starting.
I20260927 02:48:55.412550 23095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21: No bootstrap required, opened a new log
I20260927 02:48:55.413266 23020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b: Bootstrap starting.
I20260927 02:48:55.414021 23171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:55.414636 23171 log.cc:824] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:55.415560 23020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:55.416282 23171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f: No bootstrap required, opened a new log
I20260927 02:48:55.416329 23020 log.cc:824] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:55.416474 23095 raft_consensus.cc:348] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } }
I20260927 02:48:55.416694 23095 raft_consensus.cc:729] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62e08efb0704e05967763453cbb1f21, State: Initialized, Role: FOLLOWER
I20260927 02:48:55.417032 23095 consensus_queue.cc:260] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } }
I20260927 02:48:55.418059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:48:55.418272 23095 sys_catalog.cc:564] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:55.418650 23020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b: No bootstrap required, opened a new log
I20260927 02:48:55.419306 22994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1fe3b5d82e5d43b894dc2104be38a58b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46258
I20260927 02:48:55.420887 23171 raft_consensus.cc:348] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } }
I20260927 02:48:55.421136 23171 raft_consensus.cc:729] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac6a2fe388d4200a13e908c1f39738f, State: Initialized, Role: FOLLOWER
I20260927 02:48:55.421490 23171 consensus_queue.cc:260] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } }
I20260927 02:48:55.422528 23020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } }
I20260927 02:48:55.422712 23020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe3b5d82e5d43b894dc2104be38a58b, State: Initialized, Role: FOLLOWER
I20260927 02:48:55.422709 23171 sys_catalog.cc:564] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:55.422976 23020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } }
W20260927 02:48:55.423817 23192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:55.423902 23192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:55.420951 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.423991 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:55.424167 23020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:55.425364 22994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:48:55.425480 22994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:55.426762 22994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } }
W20260927 02:48:55.427656 23206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:55.427737 23206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:55.428051 23193 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.428181 23193 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:55.428150 23069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e62e08efb0704e05967763453cbb1f21"
I20260927 02:48:55.428381 23069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [term 0 FOLLOWER]: Advancing to term 1
W20260927 02:48:55.428483 22973 tablet.cc:2369] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:55.429459 23069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe3b5d82e5d43b894dc2104be38a58b in term 1.
I20260927 02:48:55.426460 23101 master_runner.cc:186] Error getting master registration for 127.0.98.62:37251: OK, instance_id { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" instance_seqno: 1790477334932201 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:48:55.429646 23101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:48:55.429845 22971 leader_election.cc:302] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe3b5d82e5d43b894dc2104be38a58b, e62e08efb0704e05967763453cbb1f21; no voters:
I20260927 02:48:55.422297 23027 master_runner.cc:186] Error getting master registration for 127.0.98.62:37251: OK, instance_id { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" instance_seqno: 1790477334932201 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:48:55.430598 23027 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:48:55.430567 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.430693 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:55.430788 23213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:55.430869 23213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [term 1 LEADER]: Becoming Leader. State: Replica: 1fe3b5d82e5d43b894dc2104be38a58b, State: Running, Role: LEADER
I20260927 02:48:55.431182 23213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } }
I20260927 02:48:55.431211 23145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fac6a2fe388d4200a13e908c1f39738f"
I20260927 02:48:55.431389 23145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:55.432647 23145 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe3b5d82e5d43b894dc2104be38a58b in term 1.
I20260927 02:48:55.433025 22994 leader_election.cc:288] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [CANDIDATE]: Term 1 election: Requested vote from peers e62e08efb0704e05967763453cbb1f21 (127.0.98.61:37361), fac6a2fe388d4200a13e908c1f39738f (127.0.98.60:36431)
I20260927 02:48:55.433724 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fe3b5d82e5d43b894dc2104be38a58b. Latest consensus state: current_term: 1 leader_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.433836 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:55.435180 23221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:55.439688 23221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:55.445675 23145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [term 1 FOLLOWER]: Refusing update from remote peer 1fe3b5d82e5d43b894dc2104be38a58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:55.446345 23195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:55.450060 23193 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fe3b5d82e5d43b894dc2104be38a58b. Latest consensus state: current_term: 1 leader_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.450181 23193 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:55.451103 23069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [term 1 FOLLOWER]: Refusing update from remote peer 1fe3b5d82e5d43b894dc2104be38a58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:55.451658 23195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:55.454867 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fe3b5d82e5d43b894dc2104be38a58b. Latest consensus state: current_term: 1 leader_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.454983 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:55.459911 23195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.460026 23195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:55.461104 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.461216 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:55.461539 23227 mvcc.cc:205] Tried to move back new op lower bound from 7333795165978624000 to 7333795165929762816. Current Snapshot: MvccSnapshot[applied={T|T < 7333795165978624000}]
I20260927 02:48:55.461894 23233 mvcc.cc:205] Tried to move back new op lower bound from 7333795165978624000 to 7333795165929762816. Current Snapshot: MvccSnapshot[applied={T|T < 7333795165978624000}]
I20260927 02:48:55.464020 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.464130 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:55.465073 23193 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe3b5d82e5d43b894dc2104be38a58b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } peers { permanent_uuid: "e62e08efb0704e05967763453cbb1f21" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "fac6a2fe388d4200a13e908c1f39738f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36431 } } }
I20260927 02:48:55.465186 23193 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:48:55.465821 23231 mvcc.cc:205] Tried to move back new op lower bound from 7333795165978624000 to 7333795165929762816. Current Snapshot: MvccSnapshot[applied={T|T < 7333795165978624000}]
I20260927 02:48:55.470386 23221 catalog_manager.cc:1347] Generated new cluster ID: 0ece1688280049e89c9c049a5e71ced7
I20260927 02:48:55.470449 23221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:55.478889 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.443384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.0.98.61:37361)
0927 02:48:55.443491 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.443495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.452312 (+ 8817us) server_negotiation.cc:407] Connection header received
0927 02:48:55.453895 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.453898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.453944 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.454004 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.455955 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.456729 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.462827 (+ 6098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.463179 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.463249 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.470327 (+ 7078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.470332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.470334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.470360 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.472049 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.476928 (+ 4879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.476932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.476934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.477031 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.477689 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.477692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.477694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.477905 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.478026 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.478616 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.478734 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:48:55.481943 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.442536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.0.98.62:37251)
0927 02:48:55.442641 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.442646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.453824 (+ 11178us) server_negotiation.cc:407] Connection header received
0927 02:48:55.454958 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.454961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.455020 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.455074 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.459833 (+ 4759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.460587 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.463853 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.464221 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.464298 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.472542 (+ 8244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.472546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.472549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.472577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.474221 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.477319 (+ 3098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.477323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.477325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.477404 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.480433 (+ 3029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.480437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.480439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.480644 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.480750 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.481682 (+ 932us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.481799 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:48:55.483517 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:55.443136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.0.98.60:36431)
0927 02:48:55.443321 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.443325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.452130 (+ 8805us) server_negotiation.cc:407] Connection header received
0927 02:48:55.452500 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.452503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.452552 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.452600 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.456858 (+ 4258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.457639 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.461690 (+ 4051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:55.462060 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:55.462135 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:55.474716 (+ 12581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:55.474720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:55.474723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:55.474751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:55.476460 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.480047 (+ 3587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.480051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.480052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.480139 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:55.481041 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:55.481045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:55.481047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:55.481255 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:55.481359 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:55.483255 (+ 1896us) server_negotiation.cc:299] Negotiation successful
0927 02:48:55.483375 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 02:48:55.485690 23221 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:55.486887 23221 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:55.502270 23221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fe3b5d82e5d43b894dc2104be38a58b: Generated new TSK 0
I20260927 02:48:55.503046 23221 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:48:55.529255 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.526353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.0.98.61:37361)
0927 02:48:55.526470 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.526475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.528238 (+ 1763us) server_negotiation.cc:407] Connection header received
0927 02:48:55.528423 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.528427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.528477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.528530 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.529128 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41364: BlockingRecv error: recv got EOF from 127.0.0.1:41364 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 02:48:55.530706 23176 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.527052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.0.98.60:36431)
0927 02:48:55.527155 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.527159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.529695 (+ 2536us) server_negotiation.cc:407] Connection header received
0927 02:48:55.530055 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.530058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.530102 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.530149 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.530598 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59896: BlockingRecv error: recv got EOF from 127.0.0.1:59896 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:48:55.531731 23024 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.527666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.0.98.62:37251)
0927 02:48:55.527768 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.527772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.529879 (+ 2107us) server_negotiation.cc:407] Connection header received
0927 02:48:55.531052 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.531056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.531102 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.531150 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.531623 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46276: BlockingRecv error: recv got EOF from 127.0.0.1:46276 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 02:48:55.535854 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.534380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.0.98.61:37361)
0927 02:48:55.534473 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.534478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.534997 (+ 519us) server_negotiation.cc:407] Connection header received
0927 02:48:55.535203 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.535206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.535273 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.535321 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.535765 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41376: BlockingRecv error: recv got EOF from 127.0.0.1:41376 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 02:48:55.540337 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.538935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.0.98.61:37361)
0927 02:48:55.539039 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.539043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.539514 (+ 471us) server_negotiation.cc:407] Connection header received
0927 02:48:55.539678 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.539682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.539723 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.539768 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.540251 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41384: BlockingRecv error: recv got EOF from 127.0.0.1:41384 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 02:48:55.545352 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.544001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.0.98.61:37361)
0927 02:48:55.544086 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.544090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.544546 (+ 456us) server_negotiation.cc:407] Connection header received
0927 02:48:55.544705 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.544708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.544749 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.544799 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.545275 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41386: BlockingRecv error: recv got EOF from 127.0.0.1:41386 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 02:48:55.552326 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.550982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.98.61:37361)
0927 02:48:55.551096 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.551100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.551549 (+ 449us) server_negotiation.cc:407] Connection header received
0927 02:48:55.551710 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.551713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.551753 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.551798 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.552249 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41400: BlockingRecv error: recv got EOF from 127.0.0.1:41400 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 02:48:55.563640 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.562186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.0.98.61:37361)
0927 02:48:55.562285 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.562289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.562791 (+ 502us) server_negotiation.cc:407] Connection header received
0927 02:48:55.562977 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.562981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.563027 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.563076 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.563551 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41404: BlockingRecv error: recv got EOF from 127.0.0.1:41404 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 02:48:55.583664 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.581926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.0.98.61:37361)
0927 02:48:55.582067 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.582071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.582659 (+ 588us) server_negotiation.cc:407] Connection header received
0927 02:48:55.582861 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.582864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.582918 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.582985 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.583555 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41410: BlockingRecv error: recv got EOF from 127.0.0.1:41410 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 02:48:55.619859 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.617955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.0.98.61:37361)
0927 02:48:55.618083 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.618089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.618718 (+ 629us) server_negotiation.cc:407] Connection header received
0927 02:48:55.618978 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.618982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.619039 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.619114 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.619733 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41414: BlockingRecv error: recv got EOF from 127.0.0.1:41414 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 02:48:55.687759 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.686473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.0.98.61:37361)
0927 02:48:55.686638 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.686643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.686880 (+ 237us) server_negotiation.cc:407] Connection header received
0927 02:48:55.687064 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.687068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.687128 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.687219 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.687655 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41420: BlockingRecv error: recv got EOF from 127.0.0.1:41420 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 02:48:55.819128 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:55.817898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.0.98.61:37361)
0927 02:48:55.818056 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:55.818060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:55.818093 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:48:55.818236 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:55.818238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:55.818287 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:55.818361 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:55.818942 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41424: BlockingRecv error: recv got EOF from 127.0.0.1:41424 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 02:48:56.078330 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:56.077277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.0.98.61:37361)
0927 02:48:56.077439 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:56.077442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:56.077554 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:48:56.077699 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:56.077702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:56.077752 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:56.077834 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:56.078234 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41432: BlockingRecv error: recv got EOF from 127.0.0.1:41432 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:48:56.425289 23192 catalog_manager.cc:1259] Loaded cluster ID: 0ece1688280049e89c9c049a5e71ced7
I20260927 02:48:56.425364 23192 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21: loading cluster ID for follower catalog manager: success
I20260927 02:48:56.427091 23192 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21: acquiring CA information for follower catalog manager: success
I20260927 02:48:56.428437 23192 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e62e08efb0704e05967763453cbb1f21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:48:56.428893 23206 catalog_manager.cc:1259] Loaded cluster ID: 0ece1688280049e89c9c049a5e71ced7
I20260927 02:48:56.428946 23206 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f: loading cluster ID for follower catalog manager: success
I20260927 02:48:56.430370 23206 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f: acquiring CA information for follower catalog manager: success
I20260927 02:48:56.431356 23206 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fac6a2fe388d4200a13e908c1f39738f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:48:56.596457 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:56.592444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.0.98.61:37361)
0927 02:48:56.592691 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:56.592696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:56.592808 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:48:56.593001 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:56.593003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:56.593125 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:56.593245 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:56.593528 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.594053 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.594936 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.595165 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.595274 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:56.595597 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:56.596064 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:56.596203 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:56.596240 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:48:56.596293 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":65,"threads_started":1}
I20260927 02:48:56.603241 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:56.599550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.0.98.60:36431)
0927 02:48:56.599790 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:56.599793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:56.599827 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:48:56.599971 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:56.599973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:56.600091 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:56.600191 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:56.600395 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.600978 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.601799 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.602005 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.602079 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:56.602375 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:56.602794 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:56.602884 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:56.602980 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:48:56.603051 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260927 02:48:56.610322 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:56.606211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.98.61:37361)
0927 02:48:56.606331 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:56.606336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:56.606475 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:48:56.606675 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:56.606679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:56.606734 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:56.606790 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:56.607207 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.607846 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.608857 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.609199 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.609300 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:56.609566 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:56.609642 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:56.609726 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:56.610110 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 02:48:56.610167 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:48:56.610684 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:56.606569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.0.98.60:36431)
0927 02:48:56.606680 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:56.606685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:56.606818 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:48:56.606928 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:56.606934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:56.606989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:56.607060 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:56.607435 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.608171 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.609259 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.609597 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.609687 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:56.610058 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:56.610157 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:56.610222 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:56.610509 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 02:48:56.610568 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:48:56.611014 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:56.606257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.0.98.62:37251)
0927 02:48:56.606600 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:56.606605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:56.606996 (+ 391us) server_negotiation.cc:407] Connection header received
0927 02:48:56.607180 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:56.607186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:56.607320 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:56.607443 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:56.607797 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.608493 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.609411 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:56.609758 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:56.609854 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:56.610242 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:56.610625 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:56.610732 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:56.610804 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:48:56.610860 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":59,"threads_started":1}
I20260927 02:48:56.614241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 22952
I20260927 02:48:56.620909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 23027
I20260927 02:48:56.627081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8LsHFz/build/debug/bin/kudu with pid 23101
2026-09-27T02:48:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3025 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6100 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (304704 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] AuthTokenIssuingTest.ChannelConfidentiality/4, where GetParam() = 32-byte object <02-00 00-00 00-00 00-00 A8-A9 06-1F 00-00 00-00 28-A5 06-1F 00-00 00-00 00-00 01-00 00-00 00-00>
2 FAILED TESTS